Unity Human Pose Estimation

Note

These steps assume you have setup the Unity SDK directions first. If you have not done so, follow those instructions to configure your Unity project.

Setup AR Foundation Objects

Fritz Pose Estimation use the ARCamera accessible through the Unity AR Foundation package. Add both AR Session and AR Session Origin objects to your Scene.

AR Foundation Objects

Setup FritzPoseController

  1. Download the starter Unity Project. to get up and running quickly.
  2. Right click on the Assets folder and select “Import Package > Custom Package” and locate the downloaded unitypackage file. This will add a FritzVisionUnity script in your Assets > Scripts folder which you can add to any AR object in the scene.
  3. Add the script as a new Component in an empty GameObject named FritzPoseController. This passes camera frames from ARKit/ARCore to the pose estimation model.
FritzVisionUnity configuration

Create a script to process camera frames

If you’d like to create your own MonoBehavior script to process the pose from the camera frame, follow the directions below.

  1. Instantiate Fritz in the Awake function

    using AI.Fritz.Vision;
    
    private void Awake()
    {
        FritzPoseManager.SetCallbackTarget("FritzPoseController");
        FritzPoseManager.SetCallbackFunctionTarget("UpdatePose");
        FritzPoseManager.Configure();
    }
    
  2. Pass frames to FritzPoseManager.

    private void Update()
    {
        if (FritzPoseManager.Processing())
        {
            return;
        }
    #if UNITY_ANDROID
    
        XRCameraImage image;
        if (!cameraManager.TryGetLatestImage(out image))
        {
            image.Dispose();
            return;
        }
    
        FritzPoseManager.ProcessPoseFromImageAsync(image);
    
        // You must dispose the CameraImage to avoid resource leaks.
        image.Dispose();
    
    #elif UNITY_IOS
        var cameraParams = new XRCameraParams
        {
            zNear = m_Cam.nearClipPlane,
            zFar = m_Cam.farClipPlane,
            screenWidth = Screen.width,
            screenHeight = Screen.height,
            screenOrientation = Screen.orientation
        };
    
        XRCameraFrame frame;
    
        if (!cameraManager.subsystem.TryGetLatestFrame(cameraParams, out frame))
        {
            return;
        }
    
        FritzPoseManager.ProcessPoseFromFrameAsync(frame);
    #endif
    }
    

    This will pass new frames to the pose model. Frames are processed asynchronously. When a frame has been successfully processed the results are passed back to Unity using a Unity Event.

    Note

    In iOS, use XRCameraFrame to capture the camera frame and then pass it to ProcessPoseFromFrameAsync. In Android, use XRCameraImage and pass it to ProcessPoseFromImageAsync.

  3. Setup Pose Handler

    You’ll need a function to receive events from the Pose Model. Unity messages are passed as strings, so we need to deserialize the detected poses into the FritzPose objects.

    public void UpdatePose(string message)
    {
        var poses = FritzPoseManager.ProcessEncodedPoses(message);
    
        foreach (FritzPose pose in poses)
        {
    
        }
    }
    

    Note

    The name of this function must be the same as specified in the Awake function above.

Human Trackables

Unity trackables allow objects to be detected an tracked in the real world. Use the FritzHumanTrackableManager to easily track the position of people in World Coordinates.

To get an updated FritzHumanTrackable:

  1. Add the FritzHumanTrackableManager and ARRaycastManager to the AR Session Origin object in your scene.
  2. Instantiate the Human Trackable Manager.
ARSessionOrigin origin = GetComponent<ARSessionOrigin>();
FritzHumanTrackableManager trackableManager = origin.GetComponent<FritzHumanTrackableManager>();
  1. In the function that receives new poses, update the trackable:
public void UpdatePose(string message)
{
    var poses = FritzPoseManager.ProcessEncodedPoses(message);

    foreach (FritzPose pose in poses)
    {
        FritzHumanTrackable trackable = trackableManager.CreateOrUpdateTrackable(0, pose);
        break;
    }
}

Note

Each Trackable is assigned a trackableID. Currently we do not generate unique IDs for multiple poses, so any identification must be done manually. We use an ID of 0 in the examples, assuming there is only one pose being updated.

Using a FritzHumanTrackable

Estimate the bounds of a person

Get the bounds of a cube enclosing a detected human body in World Coordinates to enable interactions with the Unity environment.

FritzHumanTrackable trackable = trackableManager.CreateOrUpdateTrackable(0, pose);
Transform personTransform = trackable.GetEstimatedTransform();

personTransform contains the transform the body’s inferred location in space.

Estimate positions of individual keypoints

For each keypoint, you can get the estimated 3D world coordinate. This can be used to get a rough estimate of keypoint position.

FritzHumanTrackable trackable = trackableManager.CreateOrUpdateTrackable(0, pose);
Vector3 leftShoulder = trackable.GetEstimatedPosition(FritzPoseParts.leftShoulder);

Experimental GPU Support on Android

Warning

GPU Support is experimental and may be unsupported on some devices causing unexpected crashes. Read more about GPU support for TensorFlow Lite.

On Android, Fritz users TensorFlow Lite to make pose predictions. TensorFlow Lite provides experimental GPU support to speed up predictions and improve app performance. You can use the GPU by calling the UseGPU method on the FritzPoseManager as follows:

  1. Make sure that the unity sdk version installed in unity package manager is >= 1.0.1.
  2. In the FritzVisionUnity script call the UseGpu method on FritzVisionUnity in the Update method. If you call it in the Awake method for the script, the GLContext might not be initialized yet.
private void Update()
{
    if (FritzPoseManager.Processing())
    {
        return;
    }

#if UNITY_ANDROID && !UNITY_EDITOR

    FritzPoseManager.UseGPU();

    // ...
}