Android

Note

If you haven’t set up the SDK yet, make sure to go through those directions first. You’ll need to add the Core library to the app before using the specific feature API or custom model. Follow iOS setup or Android setup directions.

1. Add the dependencies via Gradle

Add our repository in order to download the Vision API:

repositories {
    maven { url "https://raw.github.com/fritzlabs/fritz-repository/master" }
}

Include the dependencies in app/build.gradle:

dependencies {
    implementation 'ai.fritz:core:3.3.+'
    implementation 'ai.fritz:vision:3.3.+'
}

(Optional include model in your app) To include Image Labeling model with your build, then you’ll need to add the dependency as shown below. Note: This includes the model with your app when you publish it to the play store and will increase your app size.

dependencies {
  implementation 'ai.fritz:vision-image-label-model:3.3.+'
}

Now you’re ready to classify images with the Image Labeling API.

2. Get a FritzVisionLabelPredictor

In order to use the predictor, the on-device model must first be loaded. If you followed the Optional step above and included the ai.fritz:vision-image-label-model dependency, you can get a predictor to use immediately:

import ai.fritz.visionlabel.ImageLabelOnDeviceModel;
import ai.fritz.vision.imagelabeling.FritzVisionLabelPredictor;
import ai.fritz.core.FritzOnDeviceModel;

FritzOnDeviceModel imageLabelOnDeviceModel = new ImageLabelOnDeviceModel();
FritzVisionLabelPredictor predictor = FritzVision.ImageLabeling.getPredictor(imageLabelOnDeviceModel);

If you did not include the on-device model, you’ll have to load the model before you can get a predictor. To do that, you’ll use ImageLabelManagedModel object and call FritzVision.ImageLabeling.loadPredictor to start the model download.

import ai.fritz.vision.FritzVision;
import ai.fritz.vision.imagelabeling.ImageLabelManagedModel;
import ai.fritz.vision.PredictorStatusListener;
import ai.fritz.vision.imagelabeling.FritzVisionLabelPredictor;
import ai.fritz.core.FritzManagedModel;

FritzVisionLabelPredictor predictor;

FritzManagedModel managedModel = new ImageLabelManagedModel();
FritzVision.ImageLabeling.loadPredictor(managedModel, new PredictorStatusListener<FritzVisionLabelPredictor>() {
    @Override
    public void onPredictorReady(FritzVisionLabelPredictor imageLabelingPredictor) {
        Log.d(TAG, "Image Labeling predictor is ready");
        predictor = imageLabelingPredictor;
    }
});

3. Create a FritzVisionImage from an image or a video stream

To create a FritzVisionImage from a Bitmap:

FritzVisionImage visionImage = FritzVisionImage.fromBitmap(bitmap);

To create a FritzVisionImage from a media.Image object when capturing the result from a camera, first determine the orientation of the image. This will rotate the image to account for device rotation and the orientation of the camera sensor.

// Get the system service for the camera manager
final CameraManager manager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);

// Gets the first camera id
String cameraId = manager.getCameraIdList().get(0);

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
int imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId);

Finally, create the FritzVisionImage object with the rotation

FritzVisionImage visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);

4. Run prediction - Label your image

Next, pass the FritzVisionImage into the predictor in order to evaluate the image labels:

FritzVisionLabelResult labelResult = visionPredictor.predict(visionImage);

visionPredictor.predict() returns a FritzVisionLabelResult which contains several methods to access the labels.

Method Description
FritzVisionImage getOriginalImage() Get the original image passed into the predict method.
List<FritzVisionLabel> getVisionLabels() Gets a list of FritzVisionLabel objects. Each label object has getText() and getConfidence() methods.
String getResultString() Gets a string of all labels in the format “<label name>:<confidence score>”
void drawVisionImage(Canvas canvas) Draws the original image passed in to the predict method.
void drawVisionImage(Canvas canvas, Size canvasSize) Draws the original image scaled up to the target canvas size. E.g Your original image was 512x512 but you’d like to stretch it to fit a canvas size of 1024x1024.
void logResult() Uses Log.d to print out the label predictions

Advanced Options

Setting a Confidence Threshold

You may also pass options into FritzVisionLabelPredictor in order to set a confidence threshold for the results. For example, if you only want to return labels that have a confidence score of 0.7 or higher, you may specify it with the following:

// Create predictor options
FritzVisionLabelPredictorOptions options = new FritzVisionLabelPredictorOptions.Builder()
    .confidenceThreshold(0.7f)
    .build();

// Pass in the options when initializing the predictor
FritzVisionLabelPredictor predictor = FritzVision.ImageLabeling.getPredictor(imageLabelOnDeviceModel, options);

// Or pass in options on an existing predictor instance
visionPredictor.setOptions(options);

Controlling Image Preprocessing

By setting FritzVisionCropAndScale on FritzVisionLabelPredictorOptions, you can control preprocessing on the image.

There are 2 crop and scale options available:

  • FritzVisionCropAndScale.SCALE_TO_FIT (default) - Resizes the original image to fit the size of the model input while maintaining the aspect ratio. After inference, the output is scaled back to the original aspect ratio. (e.g for a model input size of 384x384, scale a 600x400 image down to fit the model size. After the model runs inference, scale the output results to match the original 600x400 aspect ratio).
  • FritzVisionCropAndScale.CENTER_CROP - Run prediction on a center cropped section of the original image passed into the predict method. (e.g if the original image is 600x400, the predictor will run on a 400x400 image)

Example usage:

FritzVisionLabelPredictorOptions options = new FritzVisionLabelPredictorOptions.Builder()
  .cropAndScaleOption(FritzVisionCropAndScale.SCALE_TO_FIT)
  .build()
FritzVisionLabelPredictor predictor = FritzVision.ImageLabeling.getPredictor(imageLabelOnDeviceModel, options);