Unity Human Pose Estimation

Note

These steps assume you have setup the Unity SDK directions first. If you have not done so, follow those instructions to configure your Unity project.

Setup AR Foundation Objects

Fritz Pose Estimation use the ARCamera accessible through the Unity AR Foundation package. Add both AR Session and AR Session Origin objects to your Scene.

AR Foundation Objects

Quickstart

To get up and running quickly, we’ve provided two example apps:

  • Basic Demo: Places a Game Object on the “Left Shoulder” keypoint.
  • Bird Perch PoseDemo: Places an animated bird on the “Left Shoulder” keypoint.

Steps:

  1. Register the example app in your Fritz account.
  2. In the Unity project’s Build Settings > Player Settings > Fritz, add the appropriate API key under iOS or Android.
  3. Build and Run.

Create a script to process camera frames

If you’d like to create your own MonoBehavior script to process the pose from the camera frame, follow the directions below.

  1. Create a new C# script and give it a name (e.g FritzVisionUnity.cs)

  2. Instantiate Fritz in the Awake function

    using AI.Fritz.Vision;
    
    private void Awake()
    {
        FritzPoseManager.SetCallbackTarget("FritzPoseController");
        FritzPoseManager.SetCallbackFunctionTarget("UpdatePose");
        FritzPoseManager.Configure();
    }
    
  3. Pass frames to FritzPoseManager.

    private void Update()
    {
        if (FritzPoseManager.Processing())
        {
            return;
        }
    #if UNITY_ANDROID
    
        XRCameraImage image;
        if (!cameraManager.TryGetLatestImage(out image))
        {
            image.Dispose();
            return;
        }
    
        FritzPoseManager.ProcessPoseFromImageAsync(image);
    
        // You must dispose the CameraImage to avoid resource leaks.
        image.Dispose();
    
    #elif UNITY_IOS
        var cameraParams = new XRCameraParams
        {
            zNear = m_Cam.nearClipPlane,
            zFar = m_Cam.farClipPlane,
            screenWidth = Screen.width,
            screenHeight = Screen.height,
            screenOrientation = Screen.orientation
        };
    
        XRCameraFrame frame;
    
        if (!cameraManager.subsystem.TryGetLatestFrame(cameraParams, out frame))
        {
            return;
        }
    
        FritzPoseManager.ProcessPoseFromFrameAsync(frame);
    #endif
    }
    

    This will pass new frames to the pose model. Frames are processed asynchronously. When a frame has been successfully processed the results are passed back to Unity using a Unity Event.

    Note

    In iOS, use XRCameraFrame to capture the camera frame and then pass it to ProcessPoseFromFrameAsync. In Android, use XRCameraImage and pass it to ProcessPoseFromImageAsync.

  4. Setup Pose Handler

    You’ll need a function to receive events from the Pose Model. Unity messages are passed as strings, so we need to deserialize the detected poses into the FritzPose objects.

    public void UpdatePose(string message)
    {
        var poses = FritzPoseManager.ProcessEncodedPoses(message);
    
        foreach (FritzPose pose in poses)
        {
    
        }
    }
    

    Note

    The name of this function must be the same as specified in the Awake function above.

  5. Add the script as a new Component in an empty GameObject in the Scene named FritzPoseController. This passes camera frames from ARKit/ARCore to the pose estimation model.

FritzVisionUnity configuration

Human Trackables

Warning

FritzHumanTrackables use hit testing to provide coarse estimates of the location of people in a scene. While they can tell you roughly how far a person is from the camera, FritzHumanTrackables are NOT suitable for true 3D pose estimation. If your use case requires 3D data for each body part and keypoint please contact us directly.

Unity trackables allow objects to be detected an tracked in the real world. Use the FritzHumanTrackableManager to easily track the position of people in World Coordinates.

To get an updated FritzHumanTrackable:

  1. Add the FritzHumanTrackableManager and ARRaycastManager to the AR Session Origin object in your scene.
  2. Instantiate the Human Trackable Manager.
ARSessionOrigin origin = GetComponent<ARSessionOrigin>();
FritzHumanTrackableManager trackableManager = origin.GetComponent<FritzHumanTrackableManager>();
  1. In the function that receives new poses, update the trackable:
public void UpdatePose(string message)
{
    var poses = FritzPoseManager.ProcessEncodedPoses(message);

    foreach (FritzPose pose in poses)
    {
        FritzHumanTrackable trackable = trackableManager.CreateOrUpdateTrackable(0, pose);
        break;
    }
}

Note

Each Trackable is assigned a trackableID. Currently we do not generate unique IDs for multiple poses, so any identification must be done manually. We use an ID of 0 in the examples, assuming there is only one pose being updated.

Using a FritzHumanTrackable

Estimate the bounds of a person

Get the bounds of a cube enclosing a detected human body in World Coordinates to enable interactions with the Unity environment.

FritzHumanTrackable trackable = trackableManager.CreateOrUpdateTrackable(0, pose);
Transform personTransform = trackable.GetEstimatedTransform();

personTransform contains the transform the body’s inferred location in space.

FritzHumanTrackable

Estimate positions of individual keypoints

For each keypoint, you can get the estimated 3D world coordinate. This can be used to get a rough estimate of keypoint position. Again, the 3D estimates provided here are coarse at best and depend on the phone’s ability to build a 3D point cloud of the scene. Typically, they are not fast enough or accurate enough for applications that require true 3D pose estimation.

FritzHumanTrackable trackable = trackableManager.CreateOrUpdateTrackable(0, pose);
Vector3 leftShoulder = trackable.GetEstimatedPosition(FritzPoseParts.leftShoulder);

Experimental GPU Support on Android

Warning

GPU Support is experimental and may be unsupported on some devices causing unexpected crashes. Read more about GPU support for TensorFlow Lite.

On Android, Fritz users TensorFlow Lite to make pose predictions. TensorFlow Lite provides experimental GPU support to speed up predictions and improve app performance. You can use the GPU by calling the UseGPU method on the FritzPoseManager as follows:

  1. Make sure that the unity sdk version installed in unity package manager is >= 1.0.1.
  2. In the FritzVisionUnity script call the UseGpu method on FritzVisionUnity in the Update method. If you call it in the Awake method for the script, the GLContext might not be initialized yet.
private void Update()
{
    if (FritzPoseManager.Processing())
    {
        return;
    }

#if UNITY_ANDROID && !UNITY_EDITOR

    FritzPoseManager.UseGPU();

    // ...
}

Troubleshooting

When I run the Unity project with the SDK, the app crashes and I’m not sure why.

  • Double check to make sure that your app is registed in Fritz and that the bundle / package Ids match when you setup the Player (Step 4 in the setup instructions)
  • For Android - Open up Android Studio or connect adb to your device and check the logs for errors.

The camera is running in the app but I don’t see any outputs on Android.

  • Make sure your gradle file is configured properly so that TensorFlow Lite models are not compressed in your Unity project under Assets/Plugins/Android/mainTemplate.gradle.
aaptOptions {
    noCompress = ['.unity3d', '.tflite', ...]
}