Android

Note

If you haven’t set up the SDK yet, make sure to go through those directions first. You’ll need to add the Core library to the app before using the specific feature or custom model libraries.

1. Add FritzVisionLabelModel via Gradle

Assuming that you’ve already included the Core SDK and the fritz repository, you’ll also want to include the object detection dependency.

In app/build.gradle:

dependencies {
    implementation 'ai.fritz:core:1.3.2'
    implementation 'ai.fritz:vision-object-model:1.3.2'
}

2. Define FritzVisionObjectPredictor

First, create a FritzVisionObjectPredictor instance:

FritzVisionObjectPredictor objectPredictor = FritzVisionObjectPredictor.getInstance(this);

Note

FritzVisionObjectPredictor.getInstance requires the calling Context as an argument (e.g Application, Activity, etc). For example, if you’re getting the predictor instance from a Fragment, you’ll want to pass in the context:

FritzVisionObjectPredictor objectPredictor = FritzVisionObjectPredictor.getInstance(this.getContext());

3. Create a FritzVisionImage from an image or a video stream

  • To create a FritzVisionImage from a Bitmap:

    FritzVisionImage visionImage = FritzVisionImage.fromBitmap(bitmap);
    
  • To create a FritzVisionImage from a media.Image object when capturing the result from a camera, first determine the orientation of the image. This will rotate the image to account for device rotation and the orientation of the camera sensor.

    // Get the system service for the camera manager
    final CameraManager manager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);
    
    // Gets the first camera id
    String cameraId = manager.getCameraIdList().get(0);
    
    // Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
    // "this" refers to the calling Context (Application, Activity, etc)
    int imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId);
    

    Finally, create the FritzVisionImage object with the rotation

    FritzVisionImage visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);
    

4. Run prediction - Detect different objects in the image

Next, convert a bitmap into a FritzVisionImage and pass the image into the predictor in order to evaluate the objects in the image:

List<FritzVisionObject> visionObjects = objectPredictor.predict(visionImage);

The predict method returns back FritzVisionObject list sorted by the confidence score. You can access the bounding box on the original image as well as the label and confidence score.

To draw the bounding boxes on the original image:

// Create a canvas to draw on.
Canvas canvas = new Canvas();

// Draw the image that was passed into the predictor
FritzVisionImage.drawOnCanvas(visionImage.getBitmap(), canvas);

// Draw the bounding boxes on the image
for (FritzVisionObject object : objects) {
    object.drawOnCanvas(getApplicationContext(), canvas);
}

Note

By default, the predictor will return back all objects with a confidence score of 0.4 or higher.

You may pass options into FritzVisionObjectPredictor in order to override these default settings. For example, if you only want to return labels that have a confidence score of 0.7 or higher and only return the top 10 objects, you may specify it with the following:

// Create predictor options
FritzVisionObjectPredictorOptions options = new FritzVisionObjectPredictorOptions.Builder()
    .confidenceThreshold(0.7f)
    .maxObjects(10)
    .build();

// Pass it into the getInstance method
FritzVisionObjectPredictor objectPredictor = FritzVisionObjectPredictor.getInstance(this, options);

// Run predict
objectPredictor.predict(visionImage);