Android

Note

If you haven’t set up the SDK yet, make sure to go through those directions first. You’ll need to add the Core library to the app before using the specific feature API or custom model. Follow iOS setup or Android setup directions.

1. Add the dependencies via Gradle

Add our repository in order to download the Vision API:

repositories {
    maven { url "https://fritz.mycloudrepo.io/public/repositories/android" }
}

Add renderscript support and include the vision dependency in app/build.gradle. Renderscript is used in order to improve image processing performance. You’ll also need to specity aaptOptions in order to prevent compressing TensorFlow Lite models.

android {
    defaultConfig {
        renderscriptTargetApi 21
        renderscriptSupportModeEnabled true
    }

    // Don't compress included TensorFlow Lite models on build.
    aaptOptions {
        noCompress "tflite"
    }
}

dependencies {
    implementation 'ai.fritz:vision:4.2.0'
}

(Optional include model in your app) To include Object Detection model with your build, then you’ll need to add the dependency as shown below. Note: This includes the model with your app when you publish it to the play store and will increase your app size.

Note

Behind the scenes, Object Detection uses a TensorFlow Lite model. In order to include this with your app, you’ll need to make sure that the model is not compressed in the APK by setting aaptOptions.

dependencies {
  implementation 'ai.fritz:vision-object-detection-model-fast:2.0.0'
}

Now you’re ready to detect objects with the Object Detection API.

2. Get a FritzVisionObjectPredictor

In order to use the predictor, the on-device model must first be loaded.

If you followed the optional step above and included the Object Detection model, you can get a predictor to use immediately:

FritzOnDeviceModel onDeviceModel = new ObjectDetectionOnDeviceModelFast();
FritzVisionObjectPredictor predictor = FritzVision.ObjectDetection.getPredictor(onDeviceModel);

If you did not include the on-device model, you’ll have to load the model before you can get a predictor. To do that, you’ll use ObjectDetectionManagedModel object and call FritzVision.ObjectDetection.loadPredictor to start the model download.

FritzVisionObjectPredictor predictor;

FritzManagedModel managedModel = new ObjectDetectionManagedModelFast();
FritzVision.ObjectDetection.loadPredictor(managedModel, new PredictorStatusListener<FritzVisionObjectPredictor>() {
    @Override
    public void onPredictorReady(FritzVisionObjectPredictor objectDetectionPredictor) {
        Log.d(TAG, "Object Detection predictor is ready");
        predictor = objectDetectionPredictor;
    }
});

3. Create a FritzVisionImage from an image or a video stream

To create a FritzVisionImage from a Bitmap:

FritzVisionImage visionImage = FritzVisionImage.fromBitmap(bitmap);
var visionImage = FritzVisionImage.fromBitmap(bitmap)

To create a FritzVisionImage from a media.Image object when capturing the result from a camera, first determine the orientation of the image. This will rotate the image to account for device rotation and the orientation of the camera sensor.

// Get the system service for the camera manager
final CameraManager manager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);

// Gets the first camera id
String cameraId = manager.getCameraIdList().get(0);

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
ImageRotation imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId);
// Get the system service for the camera manager
val manager = getSystemService(Context.CAMERA_SERVICE) as CameraManager

// Gets the first camera id
var cameraId = manager.getCameraIdList().get(0)

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
var imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId)

Finally, create the FritzVisionImage object with the rotation

FritzVisionImage visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);
val visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);

4. Run prediction - Detect different objects in the image

Next, convert a bitmap into a FritzVisionImage and pass the image into the predictor in order to evaluate the objects in the image:

FritzVisionObjectResult objectResult = objectPredictor.predict(visionImage);

The predict method returns back a FritzVisionObjectResult object that contains the following methods:

FritzVisionObjectResult methods
Type Method and Description
List<FritzVisionObject>
getObjects()
Gets a list of all objects detected.
List<FritzVisionObject>
getObjectsAboveThreshold(float confidenceThreshold)
Gets all objects detected above a certain threshold.
List<FritzVisionObject>
getVisionObjectsByClass(String labelName)
Gets all detected objects by a class name (e.g person, cat, etc)
List<FritzVisionObject>
getVisionObjectsByClass(String labelName, float labelConfidence)
Gets all detected objects by a class name (e.g person, cat, etc) and confidence score
List<FritzVisionObject>
getVisionObjectsByClasses(List<String> labelNames)
Gets all objects matching any of the provided label names
List<FritzVisionObject>
getVisionObjectsByClasses(List<String> labelNames, float labelConfidence)
Gets all objects matching any of the provided label names and confidence scores

5. Displaying the result

Create a bitmap with the bounding boxes on the original image:

// Draw the original image that was passed into the predictor
FritzVisionObject visionObject = ...;
Bitmap boundingBoxOnImage = visionImage.overlayBoundingBox(visionObject);

Draw the bounding box on a canvas:

Canvas canvas = ...;
FritzVisionObject visionObject = ...;

// Draws the bounding box on the canvas.
visionObject.draw(canvas);

Configuring the Predictor

You can configure the predictor with FritzVisionObjectPredictorOptions to return specific results that match the options given:

FritzVisionObjectPredictorOptions methods
Option Default Description
confidenceThreshold .6 Return objects detected above the confidence threshold
labels MobileNet labels a list of labels for the model.
iouThreshold .2 Intersection over union (IOU) used to detect overlapping instances and deduping detections. A higher value will requires a larger overlapping area to filter out the detection.

In order to change model performance for different devices, you may also expose the underlying TensorFlow Lite Interpreter options.

FritzVisionPredictorOptions methods
Option Default Description
useGPU false Return labels above the confidence threshold. Please note, this is an experimental option and should not be used in production apps.
useNNAPI false Uses the NNAPI for running model inference. Please note, this is an experimental option and should not be used in production apps.
numThreads <number of available threads> For CPU Only, run model inference using the specified number of threads

For more details, please visit the official TensorFlow Lite website.

  • Example:
// Create predictor options
FritzVisionObjectPredictorOptions options = new FritzVisionObjectPredictorOptions()
options.confidenceThreshold = 0.7f;
options.numThreads = 2;

// Pass in the options when initializing the predictor
FritzVisionObjectPredictor predictor = FritzVision.ObjectDetection.getPredictor(onDeviceModel, options);