Android

Note

If you haven’t set up the SDK yet, make sure to go through those directions first. You’ll need to add the Core library to the app before using the specific feature API or custom model. Follow iOS setup or Android setup directions.

1. Add the dependencies via Gradle

Add our repository in order to download the Vision API:

repositories {
    maven { url "https://fritz.mycloudrepo.io/public/repositories/android" }
}

Add renderscript support and include the vision dependency in app/build.gradle. Renderscript is used in order to improve image processing performance. You’ll also need to specity aaptOptions in order to prevent compressing TensorFlow Lite models.

android {
    defaultConfig {
        renderscriptTargetApi 21
        renderscriptSupportModeEnabled true
    }

    // Don't compress included TensorFlow Lite models on build.
    aaptOptions {
        noCompress "tflite"
    }
}

dependencies {
    implementation 'ai.fritz:vision:4.1.0'
}

(Optional include model in your app) To include Image Labeling model with your build, then you’ll need to add the dependency as shown below. Note: This includes the model with your app when you publish it to the play store and will increase your app size.

dependencies {
  implementation 'ai.fritz:vision-labeling-model-fast:2.0.0'
}

Now you’re ready to classify images with the Image Labeling API.

2. Get a FritzVisionLabelPredictor

In order to use the predictor, the on-device model must first be loaded. If you followed the Optional step above and included the ai.fritz:vision-image-label-model dependency, you can get a predictor to use immediately:

FritzOnDeviceModel imageLabelOnDeviceModel = new ImageLabelOnDeviceModelFast();
FritzVisionLabelPredictor predictor = FritzVision.ImageLabeling.getPredictor(imageLabelOnDeviceModel);

If you did not include the on-device model, you’ll have to load the model before you can get a predictor. To do that, you’ll use ImageLabelManagedModel object and call FritzVision.ImageLabeling.loadPredictor to start the model download.

FritzVisionLabelPredictor predictor;

FritzManagedModel managedModel = new ImageLabelManagedModelFast();
FritzVision.ImageLabeling.loadPredictor(managedModel, new PredictorStatusListener<FritzVisionLabelPredictor>() {
    @Override
    public void onPredictorReady(FritzVisionLabelPredictor imageLabelingPredictor) {
        Log.d(TAG, "Image Labeling predictor is ready");
        predictor = imageLabelingPredictor;
    }
});

3. Create a FritzVisionImage from an image or a video stream

To create a FritzVisionImage from a Bitmap:

FritzVisionImage visionImage = FritzVisionImage.fromBitmap(bitmap);
var visionImage = FritzVisionImage.fromBitmap(bitmap)

To create a FritzVisionImage from a media.Image object when capturing the result from a camera, first determine the orientation of the image. This will rotate the image to account for device rotation and the orientation of the camera sensor.

// Get the system service for the camera manager
final CameraManager manager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);

// Gets the first camera id
String cameraId = manager.getCameraIdList().get(0);

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
ImageRotation imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId);
// Get the system service for the camera manager
val manager = getSystemService(Context.CAMERA_SERVICE) as CameraManager

// Gets the first camera id
var cameraId = manager.getCameraIdList().get(0)

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
var imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId)

Finally, create the FritzVisionImage object with the rotation

FritzVisionImage visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);
val visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);

4. Run prediction - Label your image

Next, pass the FritzVisionImage into the predictor in order to evaluate the image labels:

FritzVisionLabelResult labelResult = visionPredictor.predict(visionImage);

visionPredictor.predict() returns a FritzVisionLabelResult which contains several methods to access the labels.

FritzVisionLabelResult methods
Method Description
List<FritzVisionLabel> getVisionLabels() Gets a list of FritzVisionLabel objects. Each label object has getText() and getConfidence() methods.
String getResultString() Creates a string with the labels and the confidence scores. (e.g cat: .35)
void logResult() Uses Log.d to print out the label predictions.

Advanced Options

Configuring the Predictor

You can configure the predictor with FritzVisionLabelPredictorOptions to return specific results that match the options given:

FritzVisionLabelPredictorOptions methods
Option Default Description
confidenceThreshold .3f Return labels above the confidence threshold
labels MobileNet labels The set of labels that the model will use.

In order to change model performance for different devices, you may also expose the underlying TensorFlow Lite Interpreter options.

FritzVisionPredictorOptions methods
Option Default Description
useGPU false Return labels above the confidence threshold. Please note, this is an experimental option and should not be used in production apps.
useNNAPI false Uses the NNAPI for running model inference. Please note, this is an experimental option and should not be used in production apps.
numThreads <number of available threads> For CPU Only, run model inference using the specified number of threads

For more details, please visit the official TensorFlow Lite website.

Example:

// Create predictor options
FritzVisionLabelPredictorOptions options = new FritzVisionLabelPredictorOptions()
options.confidenceThreshold = 0.7f;
options.numThreads = 2;

// Pass in the options when initializing the predictor
FritzVisionLabelPredictor predictor = FritzVision.ImageLabeling.getPredictor(imageLabelOnDeviceModel, options);