Android

Note

If you haven’t set up the SDK yet, make sure to go through those directions first. You’ll need to add the Core library to the app before using the specific feature API or custom model. Follow iOS setup or Android setup directions.

By identifying specific body parts and joints, the Pose Estimation API provides a way to estimate different body positions and poses in images and video.

To get started, make sure you have the model included in your app.

1. Add the dependencies via Gradle

Add our repository in order to download the Vision API:

repositories {
    maven { url "https://raw.github.com/fritzlabs/fritz-repository/master" }
}

Include the dependencies in app/build.gradle:

dependencies {
    implementation 'ai.fritz:core:3+'
    implementation 'ai.fritz:vision:3+'
}

(Optional include model in your app) To include Pose Estimation model with your build, then you’ll need to add the dependency as shown below. Note: This includes the model with your app when you publish it to the play store and will increase your app size.

Note

Behind the scenes, Pose Estimation uses a TensorFlow Lite model. In order to include this with your app, you’ll need to make sure that the model is not compressed in the APK by setting aaptOptions.

android {
  aaptOptions {
    noCompress "tflite"
  }
}

dependencies {
  implementation 'ai.fritz:vision-pose-estimation-model:3+'
}

Now you’re ready to transform images with the Pose Estimation API.

2. Get a Pose predictor

In order to use the predictor, the on-device model must first be loaded. If you followed the Optional step above and included the ai.fritz:vision-pose-estimation-model dependency, you can get a predictor to use immediately:

import ai.fritz.vision.FritzVision;
import ai.fritz.poseestimationmodel.PoseEstimationOnDeviceModel;
import ai.fritz.vision.poseestimation.FritzVisionPosePredictor;
import ai.fritz.core.FritzOnDeviceModel;
// ...

FritzOnDeviceModel onDeviceModel = new PoseEstimationOnDeviceModel();
FritzVisionPosePredictor predictor = FritzVision.PoseEstimation.getPredictor(onDeviceModel);

If you did not include the on-device model with your app, you’ll have to load the model before you can get a predictor. To do that, you’ll use PoseEstimationManagedModel and call FritzVision.PoseEstimation.loadPredictor to start the model download.

import ai.fritz.vision.FritzVision;
import ai.fritz.vision.poseestimation.PoseEstimationManagedModel;
import ai.fritz.vision.PredictorStatusListener;
import ai.fritz.vision.poseestimation.FritzVisionPosePredictor;
import ai.fritz.core.FritzManagedModel;
// ...

FritzVisionPosePredictor predictor;

FritzManagedModel managedModel = new PoseEstimationManagedModel();
FritzVision.PoseEstimation.loadPredictor(managedModel, new PredictorStatusListener<FritzVisionPosePredictor>() {
    @Override
    public void onPredictorReady(FritzVisionPosePredictor posePredictor) {
        Log.d(TAG, "Pose estimation predictor is ready");
        predictor = posePredictor;
    }
});

3. Create FritzVisionImage from an image or a video stream

To create a FritzVisionImage from a Bitmap:

FritzVisionImage visionImage = FritzVisionImage.fromBitmap(bitmap);
var visionImage = FritzVisionImage.fromBitmap(bitmap)

To create a FritzVisionImage from a media.Image object when capturing the result from a camera, first determine the orientation of the image. This will rotate the image to account for device rotation and the orientation of the camera sensor.

// Get the system service for the camera manager
final CameraManager manager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);

// Gets the first camera id
String cameraId = manager.getCameraIdList().get(0);

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
int imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId);
// Get the system service for the camera manager
val manager = getSystemService(Context.CAMERA_SERVICE) as CameraManager

// Gets the first camera id
var cameraId = manager.getCameraIdList().get(0)

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
var imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId)

Finally, create the FritzVisionImage object with the rotation

FritzVisionImage visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);
val visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);

4. Run prediction

To detect body poses in FritzVisionImage, run the following:

FritzVisionPoseResult poseResult = predictor.predict(visionImage);

The predict method returns back a FritzVisionPoseResult object that contains the following methods:

Method Description
FritzVisionImage getOriginalImage() Get the original image passed into the predict method.
void drawVisionImage(Canvas canvas) Draws the original image passed in to the predict method.
void drawVisionImage(Canvas canvas, Size targetSize) Draws the original image scaled up to the target canvas size. E.g Your original image was 512x512 but you’d like to stretch it to fit a canvas size of 1024x1024.
void drawPoses(Canvas canvas) Draw the poses (keypoints, connecting body parts) to the canvas.
void drawPoses(Canvas canvas, Size targetSize) Scale the result for the target canvas size and draw the poses (keypoints, connecting body parts).

5. Access the Pose Result

FritzVisionPoseResult contains several convenience methods to help draw the keypoints and body position.

Get a bitmap result to display in an ImageView

Using a canvas to display the result:

// Draw the result to the canvas (same size as the input image)
poseResult.drawToCanvas(canvas);
// This bitmap will have the same size as the ``FritzVisionImage`` you passed into the ``predict`` method.
Bitmap resultBitmap = poseResult.getBitmap();

// OR, you can access a scaled result
Size targetSize = new Size(2048, 2048);
Bitmap resultBitmap = poseResult.getResultBitmap(targetSize);

// Set the scaled bitmap
imageView.setBitmap(resultBitmap);

Draw the poses onto a Canvas

// [Optional] Draw the original image used to run prediction.
poseResult.drawToCanvas(canvas);

// [Optional] Draw the original image used to run prediction (scaled).
Size targetSize = new Size(2048, 2048);
poseResult.drawToCanvas(canvas, targetSize);

// Draw the pose to the canvas.
poseResult.drawPoses(canvas);

// Draw the pose to the canvas (scaled).
poseResult.drawPoses(canvas, targetSize);

Access the position of specific body keypoints

There are several body keypoints for each pose:

  • nose
  • left eye
  • right eye
  • left ear
  • right ear
  • left shoulder
  • right shoulder
  • left elbow
  • right elbow
  • left wrist
  • right wrist
  • left hip
  • right hip
  • left knee
  • right knee
  • left ankle
  • right ankle

To access each body keypoint separately:

// Get the first pose
Pose pose = poseResult.getPoses().get(0);

// Get the body keypoints
Keypoints[] keypoints = pose.getKeypoints();

// Get the name of the keypoint
String partName = keypoints[0].getPartName();
PointF keypointPoisition = keypoints[0].getPosition()

Advanced Options

Configuring the Pose predictor

You can configure the predictor with FritzVisionPosePredictorOptions to return specific results that match the options given:

Settings
cropAndScaleOption

FritzVisionCropAndScale.SCALE_TO_FIT (default)

Option for how to resize and crop the image for the model.

minPartThreshold

0.50 (default)

Minimum confidence score a keypoint must have to be included in a pose.

minPoseThreshold

0.50 (default)

Minimum confidence score a pose must have to be included in result.

maxPosesToDetect

1 (default)

Detect multiple poses in the image.

nmsRadius

20 (default)

Non-maximum suppression (NMS) distance for Part instances. Two parts

suppress each other if they are less than nmsRadius pixels away.

  • To initialize the pose predictor with options.

    FritzVisionPosePredictorOptions options = new FritzVisionPosePredictorOptions.Builder()
            .minPoseThreshold(.6f)
            .build();
    
    predictor = FritzVision.PoseEstimation.getPredictor(onDeviceModel, options);
    
  • To configure options on an existing predictor.

    FritzVisionPosePredictorOptions options = new FritzVisionPosePredictorOptions.Builder()
            .minPoseThreshold(.6f)
            .build();
    predictor.setOptions(options);