Meta Scene Understanding For Richer VR Passthrough Experiences!
Today, we will be discussing how to set up Meta's scene understanding features in Unity. These features enable the creation of more complex and scene-aware VR experiences that utilize passthrough and spatial anchors.
💡 During this video, we cover various Oculus Integration components, such as the OVR Scene Manager, OVR Semantic Classifications, and OVR Scene Model Loader. These tools will allow you to enhance your VR experiences by providing a better understanding of the virtual environment, enabling you to create more immersive and interactive content.
By the end of this tutorial, you will have a better understanding of how to use Meta's scene understanding features in Unity and how they can be used to take your VR experiences to the next level.
For a step by step written tutorial look at the steps below, or feel free to watch my Meta’s Scene Understanding video on YouTube.
Steps required to use Scene Understanding In Unity
To get stared you will first need to download the Oculus Integration package from Meta’s official website or Unity Asset Store (Affiliate Link) and be sure to use v40 or greater either for your Meta device OS as well as Oculus Integration.
On your Oculus Quest, Oculus Quest 2, or Meta Quest Pro be sure to run through the room setup process: (check out Fig 1.0 below as a reference for room setup)
Go to Settings
Experimental Features > Room Setup
Go through the process of classifying your walls, tables, lamps, windows, couches, screens, etc (this will be required for the next steps since Oculus Link doesn’t support launching Room Setup from within Link)
Create a new Unity project and import the Oculus Integration package
Be sure to enable OpenXR backend from the Unity toolbar > Oculus > Tools > OVR Utilities Plugin > Set OVR Plugin To OpenXR
Create a new Unity Scene
Add a OVRCameraRig prefab to your new scene which you can find by searching your project
Click on OVRCameraRig
Set “Anchor Support” to “Enabled”
Set “Passthrough Support” to “Supported”
Under “Insight Passthrough” set “Enable Passthrough” to true (or checked)
Add a OVRSceneManager and associate the “Plane Prefab” and “Volume Plefab” within the inspector window
Under Android > Build Settings > Player > Other Settings > Set “Color Space” to “Linear”
Scripting Backend = IL2CPP and be sure to set 64-Bit as the architecture (If you’ve any questions about this settings feel free to let me know in the comments below)
Fig 1.0 below shows room setup, passthrough demo with scene understanding, and unity view during scene understanding
How to determine if your room has been loaded by Scene Understanding ?
Meta’s OVR Scene Manager component exposes a few different events, which can assist us in understanding what’s happening with the scene understanding process. These events are defined as follows:
SceneModelLoadedSuccessfully: This event fires when the OVR Scene Manager has correctly loaded the scene definition and instantiated the prefabs for the planes and volumes. Trap it to know that the logic of the experience can now continue.
NoSceneModelToLoad: This event fires when a query load the Scene Model returns no result. It can indicate that the, user never used the Room Capture in the space they are in.
SceneCaptureReturnedWithoutError: This event will fire after the Room Capture successfully returns. It can be trapped to load thescene Model.
UnexpectedErrorWithSceneCapture: This event will fire if an error occurred while trying to send the user to Room Capture.
To bind to these events you can use this as an example:
// Bind the events associated with LoadSceneModel()
SceneManager.SceneModelLoadedSuccessfully += OnSceneModelLoadedSuccessfully;
SceneManager.NoSceneModelToLoad += OnNoSceneModelToLoad;
protected virtual void OnSceneModelLoadedSuccessfully()
{
// The scene model was captured successfully. At this point all prefabs have been instantiated.
SceneManager.Verbose?.Log(nameof(OVRSceneModelLoader),
$"{nameof(OVRSceneManager)}.{nameof(OVRSceneManager.LoadSceneModel)}() completed successfully.");
}
protected virtual void OnNoSceneModelToLoad()
{
if (_sceneCaptureRequested)
{
SceneManager.Verbose?.Log(nameof(OVRSceneModelLoader),
$"{nameof(OnSceneCaptureReturnedWithoutError)}() There is no scene model, but we have already requested scene capture once. No further action will be taken.");
}
else
{
// There's no Scene model, we have to ask the user to create one
SceneManager.Verbose?.Log(nameof(OVRSceneModelLoader),
$"{nameof(OnNoSceneModelToLoad)}() calling {nameof(OVRSceneManager)}.{nameof(OVRSceneManager.RequestSceneCapture)}()");
_sceneCaptureRequested = SceneManager.RequestSceneCapture();
}
}What are Semantic Classifications?
Meta provides various classifiers in a form of labels, each of these holds information about walls, screens, couches, tables, ceilings, and so on. All classifiers available are below and represented in C# as follows:
public static class Classification
{
public const string Floor = "FLOOR";
public const string Ceiling = "CEILING";
public const string WallFace = "WALL_FACE";
public const string Desk = "DESK";
public const string Couch = "COUCH";
public const string DoorFrame = "DOOR_FRAME";
public const string WindowFrame = "WINDOW_FRAME";
public const string Other = "OTHER";
public const string Storage = "STORAGE";
public const string Bed = "BED";
public const string Screen = "SCREEN";
public const string Lamp = "LAMP";
public const string Plant = "PLANT";
public static IReadOnlyList<string> List { get; } = new[]
{...}
}Can you do Raycast to get Semantic Classification info?
Yes, we did exactly this on our last YouTube video, reading this information in real-time can be very beneficial. In our case we used a laser to toggle game objects and mesh renderers on and off to test physics and collisions. For instance, here is the code to accomplish a raycast against game objects with OVRSemanticClassification components:
Vector3 anchorPosition = controllerAnchor.position;
Quaternion anchorRotation = controllerAnchor.rotation;
if (Physics.Raycast(new Ray(anchorPosition, anchorRotation * Vector3.forward), out hit, lineMaxLength))
{
GameObject objectHit = hit.transform.gameObject;
OVRSemanticClassification classification = objectHit?.GetComponentInParent<OVRSemanticClassification>();
if (classification != null && classification.Labels?.Count > 0)
{
displayText.text = classification.Labels[0];
}
}You can also get a copy of my the source code shown above and also through my Scene Understanding video on YouTube. To download or clone the code go to GitHub at: https://github.com/dilmerv/MetaAdvancedFeatures
Well, that concludes this post, and if you’ve any questions please let me know in the comments below, I personally feel super excited about bringing this Scene Understanding technology to Learn XR and I can’t wait to see how all of you use it!
