Android

Note

If you haven’t set up the SDK yet, make sure to go through those directions first. You’ll need to add the Core library to the app before using the specific feature API or custom model. Follow iOS setup or Android setup directions.

Fritz AI provides an Android API that you can use to partition an image into multiple segments that recognize everyday objects. Follow these simple instructions in order to bring image segmentation to your app in no time.

1. Add the dependencies via Gradle

Add our repository in order to download the Vision API:

repositories {
    maven { url "https://fritz.mycloudrepo.io/public/repositories/android" }
}

Add renderscript support and include the vision dependency in app/build.gradle. Renderscript is used in order to improve image processing performance. You’ll also need to specity aaptOptions in order to prevent compressing TensorFlow Lite models.

android {
    defaultConfig {
        renderscriptTargetApi 21
        renderscriptSupportModeEnabled true
    }

    // Don't compress included TensorFlow Lite models on build.
    aaptOptions {
        noCompress "tflite"
    }
}

dependencies {
    implementation 'ai.fritz:vision:4.1.0'
}

(Optional include model in your app) To include Image Segmentation model with your build, then you’ll need to add the dependency as shown below. Note: This includes the model with your app when you publish it to the play store and will increase your app size.

To identify people (segments that represent people are marked in Cyan).

Note

Behind the scenes, People Segmentation uses a TensorFlow Lite model. In order to include this with your app, you’ll need to make sure that the model is not compressed in the APK by setting aaptOptions.

Choose between fast, accurate, or small variants. Model variants make sure you get the right model for your use case. For more information on the tradeoffs of each variant, see Choosing a Model Variant.

Fast

  • Resolution: 384x384
  • Model Size: 26.8MB
  • Model Speed: 250ms (on Pixel)
dependencies {
    implementation 'ai.fritz:vision-people-segmentation-model-fast:2.0.0'
}

Accurate

  • Resolution: 768x768
  • Model Size: 26.8MB
  • Model Speed: 700ms (on Pixel)
dependencies {
    implementation 'ai.fritz:vision-people-segmentation-model-accurate:2.0.0'
}

Small

  • Resolution: 224x224
  • Model Size: 4MB
  • Model Speed: 250ms (on Pixel)
dependencies {
    implementation 'ai.fritz:vision-people-segmentation-model-small:2.0.0'
}

Identify Pets.

Choose between fast, accurate, or small variants. Model variants make sure you get the right model for your use case. For more information on the tradeoffs of each variant, see Choosing a Model Variant.

Fast

  • Resolution: 256x256
  • Model Size: 8.7MB
dependencies {
    implementation 'ai.fritz:vision-pet-segmentation-model-fast:2.0.0'
}

Accurate

  • Resolution: 512x512
  • Model Size: 8.7MB
dependencies {
    implementation 'ai.fritz:vision-pet-segmentation-model-fast:2.0.0'
}

Small

  • Resolution: 256x256
  • Model Size: 2.2MB
dependencies {
    implementation 'ai.fritz:vision-pet-segmentation-model-small:2.0.0'
}

Identify the sky (red).

Choose between fast, accurate, or small variants. Model variants make sure you get the right model for your use case. For more information on the tradeoffs of each variant, see Choosing a Model Variant.

Fast

  • Resolution: 256x256
  • Model Size: 8.7MB
dependencies {
    implementation 'ai.fritz:vision-sky-segmentation-model-fast:2.0.0'
}

Accurate

  • Resolution: 512x512
  • Model Size: 8.7MB
dependencies {
    implementation 'ai.fritz:vision-sky-segmentation-model-fast:2.0.0'
}

Small

  • Resolution: 256x256
  • Model Size: 2.2MB
dependencies {
    implementation 'ai.fritz:vision-sky-segmentation-model-small:2.0.0'
}

To identify the following object in your living room (with the colors that represent each segment in the final result):

  • Chair (Sandy Brown)
  • Wall (White)
  • Coffee Table (Brown)
  • Ceiling (Light Gray)
  • Floor (Dark Gray)
  • Bed (Light Blue)
  • Lamp (Yellow)
  • Sofa (Red)
  • Window (Cyan)
  • Pillow (Beige)

Choose between fast, accurate, or small variants. Model variants make sure you get the right model for your use case. For more information on the tradeoffs of each variant, see Choosing a Model Variant.

Fast

  • Resolution: 384x384
  • Model Size: 26.8MB
dependencies {
    implementation 'ai.fritz:vision-living-room-segmentation-model-fast:2.0.0'
}

Accurate

  • Resolution: 512x512
  • Model Size: 8.7MB
dependencies {
    implementation 'ai.fritz:vision-living-room-segmentation-model-accurate:2.0.0'
}

Small

  • Resolution: 256x256
  • Model Size: 2.2MB
dependencies {
    implementation 'ai.fritz:vision-living-room-segmentation-model-small:2.0.0'
}

To identify the following objects outside (with the colors that represent each segment in the final result):

  • Building / Edifice (Gray)
  • Sky (Very Light Blue)
  • Tree (Green)
  • Sidewalk / Pavement (Dark Gray)
  • Earth / Ground (Dark Green)
  • Car (Light Orange)
  • Water (Blue)
  • House (Purple)
  • Fence, Fencing (White)
  • Signboard, Sign (Pink)
  • Skyscraper (Light Gray)
  • Bridge, Span (Orange)
  • River (Light Blue)
  • Bus (Dark Orange)
  • Truck / Motortruck (dark brown)
  • Van (Light Orange)
  • Minibike / Motorbike (Black)
  • Bicycle (Dark Blue)
  • Traffic Light (Yellow)
  • Person (Cyan)

Choose between fast, accurate, or small variants. Model variants make sure you get the right model for your use case. For more information on the tradeoffs of each variant, see Choosing a Model Variant.

Fast

  • Resolution: 384x384
  • Model Size: 26.8MB
dependencies {
    implementation 'ai.fritz:vision-outdoor-segmentation-model-fast:2.0.0'
}

Accurate

  • Resolution: 512x512
  • Model Size: 8.7MB
dependencies {
    implementation 'ai.fritz:vision-outdoor-segmentation-model-accurate:2.0.0'
}

Small

  • Resolution: 256x256
  • Model Size: 2.2MB
dependencies {
    implementation 'ai.fritz:vision-outdoor-segmentation-model-small:2.0.0'
}

To identify hair in order to apply effects (coloring):

  • None
  • Hair (Red)

Choose between fast, accurate, or small variants. Model variants make sure you get the right model for your use case. For more information on the tradeoffs of each variant, see Choosing a Model Variant.

Fast

  • Resolution: 224x224
  • Model Size: 3.9MB
dependencies {
    implementation 'ai.fritz:vision-hair-segmentation-model-fast:2.0.0'
}

Accurate

  • Resolution: 512x512
  • Model Size: 8.7MB
dependencies {
    implementation 'ai.fritz:vision-hair-segmentation-model-accurate:2.0.0'
}

Small

  • Resolution: 256x256
  • Model Size: 1MB
dependencies {
    implementation 'ai.fritz:vision-hair-segmentation-model-small:2.0.0'
}

Now you’re ready to segment images with the Image Segmentation API.

2. Get a Segmentation Predictor

In order to use the predictor, the on-device model must first be loaded.

If you followed the Optional step above and included the model, you can get a predictor to use immediately:

Example for People Segmentation:

// For fast
SegmentationOnDeviceModel onDeviceModel = new PeopleSegmentationOnDeviceModelFast();

// For accurate
SegmentationOnDeviceModel onDeviceModel = new PeopleSegmentationOnDeviceModelAccurate();

// For small
SegmentationOnDeviceModel onDeviceModel = new PeopleSegmentationOnDeviceModelSmall();

FritzVisionSegmentationPredictor predictor = FritzVision.ImageSegmentation.getPredictor(onDeviceModel);
// For fast
var onDeviceModel:SegmentationOnDeviceModel = PeopleSegmentationOnDeviceModelFast()

// For accurate
var onDeviceModel:SegmentationOnDeviceModel = PeopleSegmentationOnDeviceModelAccurate()

// For small
var onDeviceModel:SegmentationOnDeviceModel = PeopleSegmentationOnDeviceModelSmall()

var predictor = FritzVision.ImageSegmentation.getPredictor(onDeviceModel)

Note

For other image segmentation models, the model class is defined with the following convention:

“<Model Type>OnDeviceModel<Model Variant>” (e.g SkyOnDeviceModelAccurate)

  • Model Types = Hair, People, LivingRoom, Outdoor, Pet, Sky
  • Model Variants = Small, Fast, Accurate

If you did not include the on-device model, you’ll have to load the model before you can get a predictor. To do that, you’ll call FritzVision.ImageSegmentation.loadPredictor to start the model download.

Example for People Segmentation:

FritzVisionSegmentationPredictor predictor;

// For fast
SegmentationManagedModel managedModel = new PeopleSegmentationManagedModelFast();

// For accurate
SegmentationManagedModel managedModel = new PeopleSegmentationManagedModelAccurate();

// For small
SegmentationManagedModel managedModel = new PeopleSegmentationManagedModelSmall();

FritzVision.ImageSegmentation.loadPredictor(managedModel, new PredictorStatusListener<FritzVisionSegmentationPredictor>() {
    @Override
    public void onPredictorReady(FritzVisionSegmentationPredictor segmentationPredictor) {
        Log.d(TAG, "Segment predictor is ready");
        predictor = segmentationPredictor;
    }
});
val predictor:FritzVisionSegmentationPredictor

// For fast
val managedModel = PeopleSegmentationManagedModelFast()

// For small
val managedModel = PeopleSegmentationManagedModelSmall()

// For accurate
val managedModel = PeopleSegmentationManagedModelAccurate()

FritzVision.ImageSegmentation.loadPredictor(managedModel, PredictorStatusListener<FritzVisionSegmentationPredictor>() {
  fun onPredictorReady(segmentationPredictor:FritzVisionSegmentationPredictor) {
    Log.d(TAG, "Segment predictor is ready")
    predictor = segmentationPredictor
  }
})

Note

For other image segmentation models, the model class is defined with the following convention:

“<Model Type>ManagedModel<Model Variant>” (e.g SkyManagedModelAccurate)

  • Model Types = Hair, People, LivingRoom, Outdoor, Pet, Sky
  • Model Variants = Small, Fast, Accurate

3. Create a FritzVisionImage from an image or a video stream

To create a FritzVisionImage from a Bitmap:

FritzVisionImage visionImage = FritzVisionImage.fromBitmap(bitmap);
var visionImage = FritzVisionImage.fromBitmap(bitmap)

To create a FritzVisionImage from a media.Image object when capturing the result from a camera, first determine the orientation of the image. This will rotate the image to account for device rotation and the orientation of the camera sensor.

// Get the system service for the camera manager
final CameraManager manager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);

// Gets the first camera id
String cameraId = manager.getCameraIdList().get(0);

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
ImageRotation imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId);
// Get the system service for the camera manager
val manager = getSystemService(Context.CAMERA_SERVICE) as CameraManager

// Gets the first camera id
var cameraId = manager.getCameraIdList().get(0)

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
var imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId)

Finally, create the FritzVisionImage object with the rotation

FritzVisionImage visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);
val visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);

4. Run prediction on FritzVisionImage

Pass your FritzVisionImage into the predictor to create masks on the original image.

FritzVisionSegmentationResult segmentationResult = predictor.predict(visionImage);
val segmentationResult = predictor.predict(visionImage);

Running predict on the image returns a FritzVisionSegmentationResult object with the following methods.

FritzVisionSegmentationResult methods
Method Description
float[][] getConfidenceScores() Get the raw confidence scores of the output. This matrix will be the same size as model output.
MaskClass[][] getMaskClassifications() Gets a grid of MaskClass objects that represent the output classification of the model (e.g MaskClass.PERSON, MaskClass.BUS, MaskClass.None). The matrix is the same size as the model output.
Bitmap buildMultiClassMask() Create a mask of the classes detected. The resulting bitmap size will be the same size as the model output.
Bitmap buildMultiClassMask(int maxAlpha, float clippingScoresAbove, float zeroingScoresBelow) Create a mask of the classes detected using the given options.
Bitmap buildSingleClassMask(MaskClass maskType) Create a mask for a given MaskClass (e.g MaskClass.PERSON)
Bitmap buildSingleClassMask(MaskClass maskType, int maxAlpha, float clippingScoresAbove, float zeroingScoresBelow) Create a mask for a given MaskClass (e.g MaskClass.PERSON) using the given options.
Bitmap buildSingleClassMask(MaskClass maskType, int maxAlpha, float clippingScoresAbove, float zeroingScoresBelow, int color) Create a mask for a given MaskClass (e.g MaskClass.PERSON) using the given options. Override the color for the MaskClass.

Calling buildSingleClassMask with clippingScoresAbove and zeroingScoresBelow arguments helps for dealing with the uncertainty of the model.

  • When clippingScoresAbove is set, any pixels with a confidence score above that threshold will have an alpha value of 255 (completely opaque).
  • When zeroingScoresBelow is set, any confidence scores below will not appear in the mask.
  • Any scores between clippingScoresAbove and zeroingScoresBelow will have a value of classProbability * 255. It’s useful to create a blur around predictions that may still contain the desired class.

5. Displaying the result

Overlay the mask on top of the original image

To view mask result, use the overlay method on the visionImage passed into the predict method.

Overlay the mask

Left: Original Image | Middle: People Segmentation mask | Right: Mask overlay

// Create a mask
Bitmap personMask = segmentationResult.buildSingleClassMask(MaskClass.PERSON);
Bitmap imageWithMask = visionImage.overlay(personMask);
val personMask = segmentationResult.buildSingleClassMask(MaskClass.PERSON);
val imageWithMask = visionImage.overlay(personMask);

Cut out the mask from the original image

To create a cut out from the original image using the mask, use the mask method on the visionImage passed into the predict method.

Create a cut out mask

Left: Original Image | Middle: Pet Segmentation mask | Right: Mask cut out

// Create a mask (max alpha value is 255, clippingScoresAbove .5f, zeroingScoresBelow .5f)
Bitmap personMask = segmentationResult.buildSingleClassMask(MaskClass.PERSON, 255, .5f, .5f);

// This image will have the same dimensions as visionImage
Bitmap imageWithMask = visionImage.mask(personMask);

// To trim the transparent pixels, set the optional trim parameter to true
Bitmap imageWithMask = visionImage.mask(personMask, true);
// Create a mask (max alpha value is 255, clippingScoresAbove .5f, zeroingScoresBelow .5f)
val personMask = segmentationResult.buildSingleClassMask(MaskClass.PERSON, 255, .5f, .5f);

// This image will have the same dimensions as visionImage
val imageWithMask = visionImage.mask(personMask);

// To trim the transparent pixels, set the optional trim parameter to true
val imageWithMask = visionImage.mask(personMask, true);

Blend the mask color with the original image :

For uses cases such as hair color changing with Hair Segmentation, you can blending the mask color with the original image. Choose from one of the following BlendModes. You may specify an alpha to apply to the mask before blending (0-255).

Blend the mask colors

Left: Original Image | Middle: Hair Segmentation mask | Right: Blended bitmap with the mask

// Hue Blend
BlendMode hueBlend = BlendMode.HUE;

// Color Blend
BlendMode colorBlend = BlendMode.COLOR;

// Soft Light Blend
BlendMode softLightBlend = BlendMode.SOFT_LIGHT;
// Hue Blend
val hueBlend = BlendMode.HUE

// Color Blend
val colorBlend = BlendMode.COLOR

// Soft Light Blend
val softLightBlend = BlendMode.SOFT_LIGHT

Set the color for the mask and then create a blended bitmap from the result.

FritzVisionSegmentationResult hairResult = hairPredictor.predict(fritzVisionImage);

Bitmap maskBitmap = hairResult.buildSingleClassMask(MaskClass.HAIR, 180, .5f, .5f, Color.BLUE);
Bitmap blendedBitmap = visionImage.blend(maskBitmap, blendMode);
val hairResult = hairPredictor.predict(fritzVisionImage)
val maskBitmap = hairResult.buildSingleClassMask(MaskClass.HAIR, 180, .5f, .5f, Color.BLUE)
val blendedBitmap = originalImage.blend(maskBitmap, blendMode)

The blendedBitmap object will have the same size as visionImage.


Advanced Options

Configuring the Predictor

You can configure the predictor with FritzVisionSegmentationPredictorOptions to return specific results that match the options given:

FritzVisionSegmentationPredictorOptions methods
Option Default Description
confidenceThreshold .3f Return labels above the confidence threshold
targetClasses Defined by the segmentation model The set of classes the model will look for

In order to change model performance for different devices, you may also expose the underlying TensorFlow Lite Interpreter options.

FritzVisionPredictorOptions methods
Option Default Description
useGPU false Return labels above the confidence threshold. Please note, this is an experimental option and should not be used in production apps.
useNNAPI false Uses the NNAPI for running model inference. Please note, this is an experimental option and should not be used in production apps.
numThreads <number of available threads> For CPU Only, run model inference using the specified number of threads

For more details, please visit the official TensorFlow Lite website.

Example:

To target only specific classes (e.g Window / Walls), create a FritzVisionSegmentationPredictorOptions object to pass into the predictor.

To initialize the options when getting the style predictor.

// List the segments to target
List<MaskClass> targetMasks = new ArrayList<>();
targetMasks.add(MaskClass.WALL);
targetMasks.add(MaskClass.WINDOW);

// Create predictor options with a confidence threshold.
// If it's below the confidence threshold, the segment will be marked
// as MaskClass.NONE.
FritzVisionSegmentationPredictorOptions options = new FritzVisionSegmentationPredictorOptions();
options.targetClasses = targetMasks;
options.confidenceThreshold = .3f;

predictor = FritzVision.ImageSegmentation.getPredictor(onDeviceModel, options);
// List the segments to target
val targetMasks = ArrayList()
targetMasks.add(MaskClass.WALL)
targetMasks.add(MaskClass.WINDOW)

// Create predictor options with a confidence threshold.
// If it's below the confidence threshold, the segment will be marked
// as MaskClass.NONE.
val options = FritzVisionSegmentationPredictorOptions();
options.targetClasses = targetMasks
options.confidenceThreshold =.3f

predictor = FritzVision.ImageSegmentation.getPredictor(onDeviceModel, options)

The resulting list will contain 3 classes: Wall, Window, and None.