Android

Fritz provides an Android API that you can use to partition an image into multiple segments that recognize everyday objects. Follow these simple instructions in order to bring image segmentation to your app in no time.

Note

If you haven’t set up the SDK yet, make sure to go through those directions first. You’ll need to add the Core library to the app before using the specific feature or custom model libraries.

1. Add FritzVisionSegmentationModel via Gradle

Assuming that you’ve already included the Core SDK and the fritz repository, you’ll also want to include the image segmentation dependency. You may choose from the following options:

In app/build.gradle:

To idenfity people (segments that represent people are marked in Cyan)

dependencies {
    implementation 'ai.fritz:core:2.0.0'
    implementation 'ai.fritz:vision-people-segment:2.0.0'
}

To idenfity the following object in your living room (with the colors that represent each segment in the final result):

  • Chair (Sandy Brown)
  • Wall (White)
  • Coffee Table (Brown)
  • Ceiling (Light Gray)
  • Floor (Dark Gray)
  • Bed (Light Blue)
  • Lamp (Yellow)
  • Sofa (Red)
  • Window (Cyan)
  • Pillow (Beige)
dependencies {
    implementation 'ai.fritz:core:2.0.0'
    implementation 'ai.fritz:vision-living-room-segment:2.0.0'
}

To idenfity the following objects outside (with the colors that represent each segment in the final result):

  • Building / Edifice (Gray)
  • Sky (Very Light Blue)
  • Tree (Green)
  • Sidewalk / Pavement (Dark Gray)
  • Earth / Ground (Dark Green)
  • Car (Light Orange)
  • Water (Blue)
  • House (Purple)
  • Fence, Fencing (White)
  • Signboard, Sign (Pink)
  • Skyscraper (Light Gray)
  • Bridge, Span (Orange)
  • River (Light Blue)
  • Bus (Dark Orange)
  • Truck / Motortruck (dark brown)
  • Van (Light Orange)
  • Minibike / Motorbike (Black)
  • Bicycle (Dark Blue)
  • Traffic Light (Yellow)
  • Person (Cyan)
dependencies {
    implementation 'ai.fritz:core:2.0.0'
    implementation 'ai.fritz:vision-outdoor-segment:2.0.0'
}

2. Get a Segmentation Predictor

Depending on the library you’ve chosen above, you’ll want to pass in your model.

For the People Segment Library:

// "this" refers to the calling Context (Application, Activity, etc)
FritzVisionSegmentPredictor segmentPredictor = new FritzVisionPeopleSegmentPredictor();

For the Living Room Segment Library:

// "this" refers to the calling Context (Application, Activity, etc)
FritzVisionSegmentPredictor segmentPredictor = new FritzVisionLivingRoomSegmentPredictor();

For the Outdoor Segment Library:

// "this" refers to the calling Context (Application, Activity, etc)
FritzVisionSegmentPredictor segmentPredictor = new FritzVisionOutdoorSegmentPredictor();

3. Create a FritzVisionImage from an image or a video stream

To create a FritzVisionImage from a Bitmap:

FritzVisionImage visionImage = FritzVisionImage.fromBitmap(bitmap);

To create a FritzVisionImage from a media.Image object when capturing the result from a camera, first determine the orientation of the image. This will rotate the image to account for device rotation and the orientation of the camera sensor.

// Get the system service for the camera manager
final CameraManager manager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);

// Gets the first camera id
String cameraId = manager.getCameraIdList().get(0);

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
int imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId);

Finally, create the FritzVisionImage object with the rotation

FritzVisionImage visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);

4. Run prediction on FritzVisionImage

Pass your FritzVisionImage into the predictor to create masks on the original image.

FritzVisionSegmentResult segmentResult = segmentPredictor.predict(visionImage);

Running predict on the image returns a FritzVisionSegmentResult object with the following methods.

Method Description
void drawVisionImage(Canvas canvas) Draws the original image passed in to the predict method.
void drawVisionImage(Canvas canvas, Size canvasSize) Draws the original image scaled up to the target canvas size. E.g Your original image was 512x512 but you’d like to stretch it to fit a canvas size of 1024x1024.
void drawAllMasks(Canvas canvas) Draws all the masks onto the canvas.
void drawAllMasks(Canvas canvas, int alpha) Draws all the masks onto the canvas. Each mask has an alpha between 0 and 255 (255 being completely opaque).
void drawAllMasks(Canvas canvas, int alpha, Size canvasSize) Draw all the masks onto the canvas with the specified alpha value and a canvas size to scale the segments. Masks are created with coordinate for the original image passed in but if you specify a canvas size, it will automatically scale the masks to fit the new specified canvas size.
FritzVisionMask findMask(MaskType maskType) Find the `FritzVisionMask object by MaskType. If none found, return null.
List<FritzVisionMask> getMasks() Gets a list of FritzVisionMask objects categorized by MaskType (e.g MaskType.PERSON, MaskType.BUS, MaskType.None)
FritzVisionImage getOriginalImage() Get the original image passed into the predict method.

Each FritzVisionMask object contains a list of FritzVisionPoint objects which indicate the pixel by pixel confidence for the given MaskType.

5. Drawing the ``FritzVisionMask`` on a canvas.

You may choose to draw all the masks identified or a specific one.

For drawing a specific mask:

Canvas canvas = new Canvas();

// Draw the original image to the canvas
FritzVisionMask personMask = segmentResult.findMask(MaskType.PERSON);

// Draw the specific mask on the canvas with alpha value 100
personMask.drawMask(canvas, 100);

For drawing all masks on the canvas (original size):

You can print the masks by themselves or on the original image.

Canvas canvas = new Canvas();

// Print the original image onto the canvas
segmentResult.drawVisionImage(canvas);

// Print the mask overlay on the canvas (without the original image)
segmentResult.drawAllMasks(canvas);

For drawing all masks on the canvas (scaled size):

Canvas canvas = new Canvas();

// Scaled size of result
Size targetSize = new Size(2048, 2048);

// Print the original image onto the canvas
segmentResult.drawVisionImage(canvas, targetSize);

// Print the mask overlay on the canvas (without the original image)
segmentResult.drawAllMasks(canvas, targetSize);

Note

To target only specific classes (e.g Window / Walls), create a FritzVisionSegmentPredictorOptions object to pass into the predictor.

To initialize the options when getting the style predictor.

// List the segments to target
List<SegmentClass> targetSegmentClasses = new ArrayList<>();
targetSegmentClasses.add(SegmentClass.WALL);
targetSegmentClasses.add(SegmentClass.WINDOW);

// Create predictor options with a confidence threshold.
// If it's below the confidence threshold, the segment will be marked
// as SegmentClass.NONE.
FritzVisionSegmentPredictorOptions options = new FritzVisionSegmentPredictorOptions.Builder()
        .targetSegmentClasses(targetSegmentClasses)
        .targetConfidenceThreshold(.3f)
        .build();

predictor = new FritzVisionLivingRoomSegmentPredictor(options);

List<FritzVisionSegment> segments = segmentPredictor.predict(visionImage);

The resulting list of segments will contain 3 classes: Wall, Window, and None.