Android

Fritz provides an Android API that you can use to partition an image into multiple segments that recognize everyday objects. Follow these simple instructions in order to bring image segmentation to your app in no time.

1. Add the dependencies via Gradle

Add our repository in order to download the Vision API:

repositories {
    maven { url "https://raw.github.com/fritzlabs/fritz-repository/master" }
}

Include the dependencies in app/build.gradle:

dependencies {
    implementation 'ai.fritz:core:3.0.2'
    implementation 'ai.fritz:vision:3.0.2'
}

(Optional include model in your app) To include Image Segmentation model with your build, then you’ll need to add the dependency as shown below. Note: This includes the model with your app when you publish it to the play store and will increase your app size.

To idenfity people (segments that represent people are marked in Cyan)

dependencies {
    implementation 'ai.fritz:vision-people-segmentation-model:3.0.2'
}

To idenfity the following object in your living room (with the colors that represent each segment in the final result):

  • Chair (Sandy Brown)
  • Wall (White)
  • Coffee Table (Brown)
  • Ceiling (Light Gray)
  • Floor (Dark Gray)
  • Bed (Light Blue)
  • Lamp (Yellow)
  • Sofa (Red)
  • Window (Cyan)
  • Pillow (Beige)
dependencies {
    implementation 'ai.fritz:vision-living-room-segmentation-model:3.0.2'
}

To idenfity the following objects outside (with the colors that represent each segment in the final result):

  • Building / Edifice (Gray)
  • Sky (Very Light Blue)
  • Tree (Green)
  • Sidewalk / Pavement (Dark Gray)
  • Earth / Ground (Dark Green)
  • Car (Light Orange)
  • Water (Blue)
  • House (Purple)
  • Fence, Fencing (White)
  • Signboard, Sign (Pink)
  • Skyscraper (Light Gray)
  • Bridge, Span (Orange)
  • River (Light Blue)
  • Bus (Dark Orange)
  • Truck / Motortruck (dark brown)
  • Van (Light Orange)
  • Minibike / Motorbike (Black)
  • Bicycle (Dark Blue)
  • Traffic Light (Yellow)
  • Person (Cyan)
dependencies {
    implementation 'ai.fritz:vision-outdoor-segmentation-model:3.0.2'
}

Now you’re ready to segment images with the Image Segmentation API.

2. Get a Segmentation Predictor

In order to use the predictor, the on-device model must first be loaded.

If you followed the Optional step above and included the model, you can get a predictor to use immediately:

import ai.fritz.peoplesegmentation.PeopleSegmentOnDeviceModel;
import ai.fritz.vision.imagesegmentation.FritzVisionSegmentPredictor;
import ai.fritz.vision.imagesegmentation.SegmentOnDeviceModel
// ...

SegmentOnDeviceModel peopleSegmentOnDeviceModel = new PeopleSegmentOnDeviceModel();
FritzVisionSegmentPredictor predictor = FritzVision.ImageSegmentation.getPredictor(peopleSegmentOnDeviceModel);
import ai.fritz.ai.fritz.fritzvisionlivingroomsegmentation.LivingRoomSegmentOnDeviceModel;
import ai.fritz.vision.imagesegmentation.FritzVisionSegmentPredictor;
import ai.fritz.vision.imagesegmentation.SegmentOnDeviceModel
// ...

SegmentOnDeviceModel livingRoomSegmentOnDeviceModel = new LivingRoomSegmentOnDeviceModel();
FritzVisionSegmentPredictor predictor = FritzVision.ImageSegmentation.getPredictor(livingRoomSegmentOnDeviceModel);
import ai.fritz.ai.fritz.fritzvisionoutdoorsegmentation.OutdoorSegmentOnDeviceModel;
import ai.fritz.vision.imagesegmentation.FritzVisionSegmentPredictor;
import ai.fritz.vision.imagesegmentation.SegmentOnDeviceModel
// ...

SegmentOnDeviceModel outdoorSegmentOnDeviceModel = new OutdoorSegmentOnDeviceModel();
FritzVisionSegmentPredictor predictor = FritzVision.ImageSegmentation.getPredictor(outdoorSegmentOnDeviceModel);

If you did not include the on-device model, you’ll have to load the model before you can get a predictor. To do that, you’ll call FritzVision.ImageLabeling.loadPredictor to start the model download.

import ai.fritz.vision.imagesegmentation.PeopleSegmentManagedModel;
import ai.fritz.vision.imagesegmentation.FritzVisionSegmentPredictor;
import ai.fritz.vision.imagesegmentation.SegmentManagedModel
// ...

FritzVisionSegmentPredictor predictor;

SegmentManagedModel managedModel = new PeopleSegmentManagedModel();
FritzVision.ImageSegmentation.loadPredictor(managedModel, new PredictorStatusListener<FritzVisionSegmentPredictor>() {
    @Override
    public void onPredictorReady(FritzVisionSegmentPredictor segmentPredictor) {
        Log.d(TAG, "Segment predictor is ready");
        predictor = segmentPredictor;
    }
});
import ai.fritz.vision.imagesegmentation.LivingRoomSegmentManagedModel;
import ai.fritz.vision.imagesegmentation.FritzVisionSegmentPredictor;
import ai.fritz.vision.imagesegmentation.SegmentManagedModel
// ...

FritzVisionSegmentPredictor predictor;

SegmentManagedModel managedModel = new LivingRoomSegmentManagedModel();
FritzVision.ImageSegmentation.loadPredictor(managedModel, new PredictorStatusListener<FritzVisionSegmentPredictor>() {
    @Override
    public void onPredictorReady(FritzVisionSegmentPredictor segmentPredictor) {
        Log.d(TAG, "Segment predictor is ready");
        predictor = segmentPredictor;
    }
});
import ai.fritz.vision.imagesegmentation.OutdoorSegmentManagedModel;
import ai.fritz.vision.imagesegmentation.FritzVisionSegmentPredictor;
import ai.fritz.vision.imagesegmentation.SegmentManagedModel
// ...

FritzVisionSegmentPredictor predictor;

SegmentManagedModel managedModel = new OutdoorSegmentManagedModel();
FritzVision.ImageSegmentation.loadPredictor(managedModel, new PredictorStatusListener<FritzVisionSegmentPredictor>() {
    @Override
    public void onPredictorReady(FritzVisionSegmentPredictor segmentPredictor) {
        Log.d(TAG, "Segment predictor is ready");
        predictor = segmentPredictor;
    }
});

3. Create a FritzVisionImage from an image or a video stream

To create a FritzVisionImage from a Bitmap:

FritzVisionImage visionImage = FritzVisionImage.fromBitmap(bitmap);

To create a FritzVisionImage from a media.Image object when capturing the result from a camera, first determine the orientation of the image. This will rotate the image to account for device rotation and the orientation of the camera sensor.

// Get the system service for the camera manager
final CameraManager manager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);

// Gets the first camera id
String cameraId = manager.getCameraIdList().get(0);

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
int imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId);

Finally, create the FritzVisionImage object with the rotation

FritzVisionImage visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);

4. Run prediction on FritzVisionImage

Pass your FritzVisionImage into the predictor to create masks on the original image.

FritzVisionSegmentResult segmentResult = predictor.predict(visionImage);

Running predict on the image returns a FritzVisionSegmentResult object with the following methods.

Method Description
void drawVisionImage(Canvas canvas) Draws the original image passed in to the predict method.
void drawVisionImage(Canvas canvas, Size canvasSize) Draws the original image scaled up to the target canvas size. E.g Your original image was 512x512 but you’d like to stretch it to fit a canvas size of 1024x1024.
void drawAllMasks(Canvas canvas) Draws all the masks onto the canvas.
void drawAllMasks(Canvas canvas, int alpha) Draws all the masks onto the canvas. Each mask has an alpha between 0 and 255 (255 being completely opaque).
void drawAllMasks(Canvas canvas, int alpha, Size canvasSize) Draw all the masks onto the canvas with the specified alpha value and a canvas size to scale the segments. Masks are created with coordinate for the original image passed in but if you specify a canvas size, it will automatically scale the masks to fit the new specified canvas size.
FritzVisionMask findMask(MaskType maskType) Find the `FritzVisionMask object by MaskType. If none found, return null.
List<FritzVisionMask> getMasks() Gets a list of FritzVisionMask objects categorized by MaskType (e.g MaskType.PERSON, MaskType.BUS, MaskType.None)
FritzVisionImage getOriginalImage() Get the original image passed into the predict method.

Each FritzVisionMask object contains a list of FritzVisionPoint objects which indicate the pixel by pixel confidence for the given MaskType.

5. Drawing the ``FritzVisionMask`` on a canvas.

You may choose to draw all the masks identified or a specific one.

For drawing a specific mask:

Using a canvas to display the result:

// Draw the original image
segmentResult.drawVisionImage(canvas);

// Draw the original image to the canvas
FritzVisionMask personMask = segmentResult.findMask(MaskType.PERSON);

// Draw the specific mask on the canvas with alpha value 100
personMask.drawMask(canvas, 100);

Using an ImageView to display the result:

// Create a mutable bitmap from the original image
Bitmap originalBitmap = segmentResult.getOriginalImage().getBitmap();
Bitmap mutableBitmap = originalBitmap.copy(Bitmap.Config.ARGB_8888, true);

// Creating a canvas to add the mask to the mutable bitmap.
Canvas canvas = new Canvas(mutableBitmap);

// Draw the original image to the canvas
FritzVisionMask personMask = segmentResult.findMask(MaskType.PERSON);

// Draw the specific mask on the canvas with alpha value 100
personMask.drawMask(canvas, 100);

// Set the bitmap for the image view
imageView.setImageBitmap(mutableBitmap);

For drawing all masks on the canvas (original size):

You can print the masks by themselves or on the original image.

Using a canvas to display the result:

// Draw the original image
segmentResult.drawVisionImage(canvas);

// Draw the masks on the canvas (without the original image)
segmentResult.drawAllMasks(canvas);

Using an ImageView to display the result:

Bitmap resultBitmap = segmentResult.getResultBitmap();

// Set the bitmap for the image view
imageView.setImageBitmap(resultBitmap);

For drawing all masks on the canvas (scaled size):

Using an ImageView to display the result:

Using a canvas to display the result:

// Scaled size of result
Size targetSize = new Size(2048, 2048);

// Draw the original image
segmentResult.drawVisionImage(canvas, targetSize);

// Draw the masks on the canvas
segmentResult.drawAllMasks(canvas, targetSize);
// Scaled size of result
Size targetSize = new Size(2048, 2048);

Bitmap resultBitmap = segmentResult.getResultBitmap(targetSize);

// Set the bitmap for the image view
imageView.setImageBitmap(resultBitmap);

Advanced Options

For targeting specific classes:

To target only specific classes (e.g Window / Walls), create a FritzVisionSegmentPredictorOptions object to pass into the predictor.

To initialize the options when getting the style predictor.

// List the segments to target
List<SegmentClass> targetSegmentClasses = new ArrayList<>();
targetSegmentClasses.add(SegmentClass.WALL);
targetSegmentClasses.add(SegmentClass.WINDOW);

// Create predictor options with a confidence threshold.
// If it's below the confidence threshold, the segment will be marked
// as SegmentClass.NONE.
FritzVisionSegmentPredictorOptions options = new FritzVisionSegmentPredictorOptions.Builder()
        .targetSegmentClasses(targetSegmentClasses)
        .targetConfidenceThreshold(.3f)
        .build();

predictor = FritzVision.ImageSegmentation.getPredictor(onDeviceModel, options);

The resulting list of segments will contain 3 classes: Wall, Window, and None.

Controlling Image Preprocessing

By setting FritzVisionCropAndScale on FritzVisionSegmentPredictorOptions, you can control preprocessing on the image.

There are 2 crop and scale options available:

  • FritzVisionCropAndScale.SCALE_TO_FIT (default) - Resizes the original image to fit the size of the model input while maintaining the aspect ratio. After inference, the output is scaled back to the original aspect ratio. (e.g for a model input size of 384x384, scale a 600x400 image down to fit the model size. After the model runs inference, scale the output results to match the original 600x400 aspect ratio).
  • FritzVisionCropAndScale.CENTER_CROP - Run prediction on a center cropped section of the original image passed into the predict method. (e.g if the original image is 600x400, the predictor will run on a 400x400 image)

Example usage:

FritzVisionSegmentPredictorOptions options = new FritzVisionSegmentPredictorOptions.Builder()
  .cropAndScaleOption(FritzVisionCropAndScale.SCALE_TO_FIT)
  .build()
predictor = FritzVision.ImageSegmentation.getPredictor(onDeviceModel, options);