Magic Leap 2 INPUT API Features (Controller & Head Pose)
Over the last few days, I've been learning how the Magic Leap 2 handles input for the controller, head pose, hand tracking, and lastly eye tracking. For now, let's focus on the controller and head pose, but later on, I will do a follow-up post about my experience with the other input options.
If you've used Unity's new input system, which is not so new anymore, then you will feel just right at home when handling input when developing for the ML2. So, ML2 uses input action assets, which hold all the bindings and schemes for all of the input options. You can find the existing input action asset by going to your Unity editor Project Tab > Packages > Magic Leap SDK > Runtime > Deprecated > MagicLeapInputs.inputactions (not sure why it is in a deprecated folder since the XR Rig -> Game Controller has bindings to it - I will come back to this post in the future if this changes).
How Do We Capture Controller Input?
When needing to capture input wether you need a trigger, bumper, menu, or other button from the controller, you will need to create two instances and enable the controller actions as follows.
MagicLeapInputs magicLeapInputs = new MagicLeapInputs();
magicLeapInputs.Enable();
MagicLeapInputs.ControllerActions controllerActions = new MagicLeapInputs.ControllerActions(magicLeapInputs);Magic Leap 2 Input Running On A Physical Device
The 1st line creates an instance of MagicLeapInputs (the input actions asset file we talked about in the intro). The word 'new' instantiates this object into memory. The 2nd line is a standard from the Input System. Normally, once you have your input instance, you have to enable it before you can use it, which is what the `Enable()` method does. Lastly, the 3rd line holds all the InputAction and callbacks, which requires that we pass the MagicLeapInputs instance before accessing any actions or setting up listeners.
Let's say you wanted to set up an action when someone holds the trigger button and perhaps also the bumper. Then your code will look like this:
void Start()
{
MagicLeapInputs magicLeapInputs = new MagicLeapInputs();
magicLeapInputs.Enable();
MagicLeapInputs.ControllerActions controllerActions = new MagicLeapInputs.ControllerActions(magicLeapInputs);
controllerActions.Trigger.performed += TriggerPerformed;
controllerActions.Bumper.performed += BumperPerformed;
}
private void TriggerPerformed(InputAction.CallbackContext obj)
{
Logger.Instance.LogInfo($"Trigger Performed");
}
private void BumperPerformed(InputAction.CallbackContext obj)
{
Logger.Instance.LogInfo($"Bumper Performed");
}What if you wanted to read information directly without having to create a listener, as coded previously with `Trigger.performed` and `Bumper.performed`? Or how about handling different data types? A controller button could perhaps return true or false, or a float, or even a `Vector3`. Well, initially, I had those questions as well, and here's an example of how to achieve it:
private void Update()
{
if (controllerActions.IsTracked.IsPressed())
{
// reading the values from InputActions
var controllerPosition = controllerActions.Position.ReadValue<Vector3>();
var controllerRotation = controllerActions.Rotation.ReadValue<Quaternion>();
Logger.Instance.LogInfo($"Controller Position: {controllerPosition}");
Logger.Instance.LogInfo($"Controller Rotation: {controllerRotation}");
Logger.Instance.LogInfo($"Controller Bumper Action: {controllerActions.Bumper.inProgress}");
Logger.Instance.LogInfo($"Controller Trigger Action: {controllerActions.Trigger.inProgress}");
Logger.Instance.LogInfo($"Controller Acceleration: {controllerActions.Acceleration.ReadValue<Vector3>()}");
Logger.Instance.LogInfo($"Controller Touchpad Position: {controllerActions.TouchpadPosition.ReadValue<Vector2>()}");
Logger.Instance.LogInfo($"Controller Touchpad Force: {controllerActions.TouchpadForce.ReadValue<float>()}");
}
}Fig 1.0 - Magic Leap 2 Controller Input Table
A different question you may have is how to access the corresponding action data during a callback. Remember, we used `InputAction.CallbackContext` as an argument for our listeners. Well, that type holds a generic method called `ReadValue<T>`, in which 'T' (a Generic) can be replaced with the data type you are targeting. It is also important not to mistakenly bind to the incorrect data type, as doing so will result in a runtime exception and therefore crash your application (I recommend looking at Fig 1.0 to determine the data types expected for each action). Let's look at the coding example below, which I created during my latest YouTube video.
private void TriggerPerformed(InputAction.CallbackContext obj)
{
lastGeneratedRandomColor = Random.ColorHSV(0f, 1f, 1f, 1f, 0.5f, 1f);
controllerArea.GetComponent<Renderer>().material.color = lastGeneratedRandomColor;
var triggerValue = obj.ReadValue<float>();
Logger.Instance.LogInfo($"Trigger Performed {triggerValue}");
}
private void BumperPerformed(InputAction.CallbackContext obj)
{
var randomScale = Random.Range(0.05f, 0.25f);
controllerArea.transform.localScale = new Vector3(randomScale, randomScale, randomScale);
Logger.Instance.LogInfo($"Bumper Performed");
}Both of the examples above have an "InputAction.CallbackContext" as the method arguments. For "TriggerPerformed," you will see `obj.ReadValue<float>`, in which case we're asking for the trigger value of a float as the user presses the trigger. This is an analog control type, and it will change from 0.0f to 1.0f depending on how far the user presses the trigger button. In the case of "BumperPerformed," we ignore the "InputAction.CallbackContext" argument completely and just use the execution to randomize the size of the control area.
How Do We Capture Head Pose Input?
Fig 1.1 - Head Input Manager Component
Currently, Magic Leap SDK uses the Input System Tracked Pose Driver component to apply rotation and movement to the camera and game controller. You can use that component with the XR Rig, which is what happens by default. However, you could also access head pose information directly if that was required. For instance, in my video, I used the head pose data to allow the UI to follow me around and also apply the proper rotation based on my head pose. See the code example below, as well as Fig 1.1 for an example of the HeadInputManager.cs created during my latest YouTube video.
void Start()
{
headposePositionInputAction.action.Enable();
headposePositionInputAction.action.performed += PositionChanged;
headposeRotationInputAction.action.Enable();
headposeRotationInputAction.action.performed += RotationChanged;
}
private void PositionChanged(InputAction.CallbackContext obj)
{
var headposePosition = obj.ReadValue<Vector3>();
objectToControl.transform.position = Vector3.Lerp(objectToControl.transform.position,
headposePosition + headposeOffset, smoothSpeed * Time.deltaTime);
}
private void RotationChanged(InputAction.CallbackContext obj)
{
var headposeRotation = obj.ReadValue<Quaternion>();
headposeRotation.y *= -1;
headposeRotation.x *= -1;
objectToControl.transform.rotation = Quaternion.Slerp(objectToControl.transform.rotation,
headposeRotation, smoothSpeed * Time.deltaTime);
}Magic Leap 2 Application Simulator
Magic Leap App Simulator for Unity
Magic Leap offers a tool called the "Application Simulator," which could be used as standalone. This means there is an entire tool available outside of Unity to simulate your ML2 headset, as well as another package in Unity that integrates the entire "Application Simulator" in Unity. This is achieved by using custom inspector windows developed by Magic Leap. Honestly, it's a super powerful suite of features that allows you to simulate nearly all input features without having to deploy. And YES, I used it a lot when testing input, so I recommend you download it from the Magic Leap Hub. Make sure to select "Magic Leap App Simulator for Unity" from the ML Hub package manager.
Helpful Magic Leap 2 Input Resources
Here are a couple of resources that I found very helpful when testing these features:
Controller api overview
Application simulator action bindings
Helpful ways to read input within callbacks
Controller gestures overview
Well, that was pretty fun to test Magic Leap 2 Input features, and knowing that it supports the new Input System (NOT TOO NEW lol) makes it all worth my time. I have a lot to keep exploring, including hand tracking and eye tracking, which I will write about once I launch my next YouTube video. Also, I did some experimenting with touchpad gestures, so I recommend watching the video as I forgot to add it to this post, but the video visuals will make it all worth your time!
Cheers, everyone, and thanks for reading this post.
