ARCore Scene Semantics for Augmented Reality Development Is Here!
With the exponential growth of the Extended Reality (XR) landscape, Google's ARCore is pushing the boundaries of Augmented Reality (AR) development. One of the latest features that got me excited was the release of 'Scene Semantics' – a ARCore feature that pushes the capabilities of AR applications, making them more aware of their surroundings. Today, let's take a deeper into the semantic segmentation feature of ARCore and how to build applications using it. I also recommend watching my latest YouTube video, where I cover these features in a practical manner.
Why should you use ARCore Scene Semantics?
Semantic segmentation is a very helpful feature that can transform the way AR applications interpret their environment. It essentially enables applications to identify key components in the real world, like the sky, terrain, vehicles, people, trees, objects, and many more aspects, thereby creating more immersive and interactive experiences for the users.
(Take a look at the images below which demonstrate how I used this technology on my last ARCore Scene Semantics API YouTube video)
But How Do You Integrate Scene Semantics Into Your AR Applications?
Let's break it down and guide you through building a prototype from scratch.
Firstly, it's crucial to ensure that your device supports semantic segmentation. This doesn't require explicit user permissions, but rather a simple check to ensure that your device is compatible with the Scene Semantics feature from the ARCore API. As developers, it is very important to confirm this compatibility to ensure your application can successfully leverage the Scene Semantics capabilities.
In Unity - simply enable “Semantics” from the ARCore Extensions Config
Once you've confirmed device compatibility, the next step is to access the texture provided by ARCore. This texture contains valuable data about the segmented parts of the scene. Leveraging this information, developers can programmatically highlight specific segments, add interactive elements, or even alter the visual properties of various scene parts, thus creating a more engaging AR experience.
In Unity - ARCore provides an easy way to request a semantic texture and also a semantic confidence texture. Take a look at the code below as a starting point, TryGetSemanticTexture and TryGetSemanticConfidenceTexture are the two methods you want to look at and you can review ConvertR8ToRGBA32Flipped from GitHub.
For a comprehensive look at how to integrate ARCore Scene Semantics in Unity take a look at my Unity Project on Github.
private bool RequestSemanticTexture(ref Texture2D result)
{
if (!semanticManager.TryGetSemanticTexture(ref inputTexture))
return false;
if (showSemanticConfidenceImage.isOn)
{
if (semanticManager.TryGetSemanticConfidenceTexture(ref inputConfidenceTexture))
semanticConfidenceImage.texture = inputConfidenceTexture;
}
ConvertR8ToRGBA32Flipped(ref inputTexture, ref result);
return true;
}The beauty of scene semantics is not just in identifying objects but in the way it facilitates interaction with these objects. With the right coding, you can manipulate how your AR app interacts with these elements, creating dynamic experiences that are not just visually stunning but also deeply engaging.
To illustrate, imagine developing an app where users can customize their environment - changing the color of the sky, adding digital flora, or spawning fantastical creatures that interact with real-world elements. All this is possible with ARCore's Scene Semantics and all the additional features such as: plane detection, point clouds, geospatial tiles, and much more available now to AR developers.
ARCore Scene Semantics Compatibility?
To utilize the Scene Semantics API, it's important to make sure your device is compatible. The API shares the same list of supported devices as the Depth API. For the most current list of compatible devices, we recommend referring to the ARCore supported devices page.
Identifying Supported Use Cases
Understanding the appropriate use cases for the Scene Semantics API is also crucial to its successful application. The API is primarily designed for the following scenarios:
Outdoor Scenes: The API is optimized for outdoor scenes. While it offers a wealth of possibilities for open-air scenarios, it's worth noting that its use is not intended for indoor environments.
Portrait Orientation: Another key aspect to consider is that the Scene Semantics API is best used in the device's default orientation mode, typically portrait. Although the API is functional in landscape mode, the quality of semantic labels may not be as accurate.
Platform Compatibility: Lastly, it's crucial to mention that the Scene Semantics API is currently available only for Android devices. This platform-specific availability means you'll need an Android device to take advantage of this powerful feature.
Niantic Lightship ARDK - Device requirements (iOS and Android) Fig 1.0
At this point, you may be wondering, "Why isn't iOS included?" That's a valid question. At present, ARCore's Scene Semantics feature does not support iOS. However, I'm confident that this will change in the near future. In the meantime, I recommend exploring Niantic’s Lightship ARDK. It offers Semantic Segmentation features that support both iOS and Android devices, thereby providing a comprehensive solution for AR development across both platforms.
In conclusion, the release of Scene Semantics in ARCore has unlocked new possibilities in the realm of AR development. Its power lies in augmenting the awareness of AR applications, making them more realistic, engaging, and contextually aware. Whether you're a seasoned AR developer or just starting your journey, now is the time to leverage this innovative feature to push the boundaries of what's possible in AR. Happy coding!
