Android

Note

If you haven’t set up the SDK yet, make sure to go through those directions first. You’ll need to add the Core library to the app before using the specific feature or custom model libraries.

1. Add FritzVisionLabelModel via Gradle

Assuming that you’ve already included the Core SDK and the fritz repository, you’ll also want to include the object detection dependency.

In app/build.gradle:

dependencies {
    implementation 'ai.fritz:core:2.1.0'
    implementation 'ai.fritz:vision-object:2.1.0'
}

2. Define FritzVisionObjectPredictor

First, create a FritzVisionObjectPredictor instance:

FritzVisionObjectPredictor objectPredictor = new FritzVisionObjectPredictor();

3. Create a FritzVisionImage from an image or a video stream

To create a FritzVisionImage from a Bitmap:

FritzVisionImage visionImage = FritzVisionImage.fromBitmap(bitmap);

To create a FritzVisionImage from a media.Image object when capturing the result from a camera, first determine the orientation of the image. This will rotate the image to account for device rotation and the orientation of the camera sensor.

// Get the system service for the camera manager
final CameraManager manager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);

// Gets the first camera id
String cameraId = manager.getCameraIdList().get(0);

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
int imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId);

Finally, create the FritzVisionImage object with the rotation

FritzVisionImage visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);

4. Run prediction - Detect different objects in the image

Next, convert a bitmap into a FritzVisionImage and pass the image into the predictor in order to evaluate the objects in the image:

FritzVisionObjectResult objectResult = objectPredictor.predict(visionImage);

The predict method returns back a FritzVisionObjectResult object that contains the following methods:

Method Description
void drawVisionImage(Canvas canvas) Draws the original image passed in to the predict method.
void drawVisionImage(Canvas canvas, Size canvasSize) Draws the original image scaled up to the target canvas size. E.g Your original image was 512x512 but you’d like to stretch it to fit a canvas size of 1024x1024.
void drawBoundingBoxes(Canvas canvas) Draws the bounding boxes on a canvas. The bounding box location is set for the original image.
void drawBoundingBoxes(Canvas canvas, Size canvasSize) Draws the bounding boxes on a canvas. The bounding boxes are set for the target canvas size.
List<FritzVisionObject> getVisionObjects() Gets a list of FritzVisionObject objects which have getVisionLabel() and getBoundingBox() methods.
FritzVisionImage getOriginalImage() Get the original image passed into the predict method.

Note

By default, the predictor will return back all objects with a confidence score of 0.4 or higher.

You may pass options into FritzVisionObjectPredictor in order to override these default settings. For example, if you only want to return labels that have a confidence score of 0.7 or higher and only return the top 10 objects, you may specify it with the following:

// Create predictor options
FritzVisionObjectPredictorOptions options = new FritzVisionObjectPredictorOptions.Builder()
    .confidenceThreshold(0.7f)
    .maxObjects(10)
    .build();

// Pass options during instatiation
FritzVisionObjectPredictor objectPredictor = new FritzVisionObjectPredictor(options);

// Pass options to an existing predictor
objectPredictor.setOptions(options);

4. Draw the bounding boxes on the canvas.

To draw the bounding boxes on the original image.

Using a canvas to display the result:

// Draw the original image that was passed into the predictor
objectResult.drawVisionImage(canvas);

// Draw the bounding boxes on the canvas
objectResult.drawBoundingBoxes(canvas);

Using an ImageView to display the result:

// Create a mutable bitmap from the original image
Bitmap originalBitmap = segmentResult.getOriginalImage().getBitmap();
Bitmap mutableBitmap = originalBitmap.copy(Bitmap.Config.ARGB_8888, true);

// Creating a canvas to add the mask to the mutable bitmap.
Canvas canvas = new Canvas(mutableBitmap);

// Draw the bounding boxes on the canvas
objectResult.drawBoundingBoxes(canvas);

// Set the bitmap for the image view
imageView.setImageBitmap(mutableBitmap);

To draw the bounding boxes on a scaled image.

If your model ran object prediction on a 1024x1024 image but you’d like to draw a 2048x2048 canvas, you can specify a targetSize as shown below.

Using a canvas to display the result:

// Target canvas size
Size targetSize = new Size(2048, 2048);

// Draw the original image
objectResult.drawVisionImage(canvas, targetSize)

// Draw the bounding boxes on the canvas
objectResult.drawBoundingBoxes(canvas, targetSize);

Using an ImageView to display the result:

// Target canvas size
Size targetSize = new Size(2048, 2048);

// Create a mutable bitmap from the original image
Bitmap originalBitmap = segmentResult.getOriginalImage().getBitmap();
Bitmap scaledBitmap = BitmapUtils.scale(originalBitmap, targetSize.getWidth(), targetSize.getHeight());
Bitmap mutableBitmap = scaledBitmap.copy(Bitmap.Config.ARGB_8888, true);

// Creating a canvas to add the mask to the mutable bitmap.
Canvas canvas = new Canvas(mutableBitmap);

// Draw the scaled bounding boxes on the same canvas
objectResult.drawBoundingBoxes(canvas, targetSize);

// Set the bitmap for the image view
imageView.setImageBitmap(mutableBitmap);

Advanced Options

Controlling Image Preprocessing

By setting FritzVisionCropAndScale on FritzVisionObjectPredictorOptions, you can control preprocessing on the image.

There are 2 crop and scale options available:

  • FritzVisionCropAndScale.SCALE_TO_FIT (default) - Resizes the original image to fit the size of the model input while maintaining the aspect ratio. After inference, the output is scaled back to the original aspect ratio. (e.g for a model input size of 384x384, scale a 600x400 image down to fit the model size. After the model runs inference, scale the output results to match the original 600x400 aspect ratio).
  • FritzVisionCropAndScale.CENTER_CROP - Run prediction on a center cropped section of the original image passed into the predict method. (e.g if the original image is 600x400, the predictor will run on a 400x400 image)

Example usage:

FritzVisionObjectPredictorOptions options = new FritzVisionObjectPredictorOptions.Builder()
  .cropAndScaleOption(FritzVisionCropAndScale.SCALE_TO_FIT)
  .build()
predictor = new FritzVisionObjectPredictorOptions(options);