Android

Note

If you haven’t set up the SDK yet, make sure to go through those directions first. You’ll need to add the Core library to the app before using the specific feature API or custom model. Follow iOS setup or Android setup directions.

Fritz AI provides an Android API that you can use to transform images or live video into beautiful works of art. You can choose to stylize an image from the photo gallery or transform live video captured by the device’s camera. Follow these simple instructions in order to bring style transfer to your app in no time.

Use Painting Styles

Follow these directions in order to apply one of the pre-trained styles to your app.

1. Add the dependencies via Gradle

Add our repository in order to download the Vision API:

repositories {
    maven { url "https://fritz.mycloudrepo.io/public/repositories/android" }
}

Add renderscript support and include the vision dependency in app/build.gradle. Renderscript is used in order to improve image processing performance. You’ll also need to specity aaptOptions in order to prevent compressing TensorFlow Lite models.

android {
    defaultConfig {
        renderscriptTargetApi 21
        renderscriptSupportModeEnabled true
    }

    // Don't compress included TensorFlow Lite models on build.
    aaptOptions {
        noCompress "tflite"
    }
}

dependencies {
    implementation 'ai.fritz:vision:4.1.0'
}

(Optional include model in your app) To include Style Transfer model with your build, then you’ll need to add the dependency as shown below. Note: This includes the model with your app when you publish it to the play store and will increase your app size.

dependencies {
  implementation 'ai.fritz:vision-style-painting-models:2.0.0'
}

Now you’re ready to transform images with the Style Transfer API.

2. Get a style predictor

In order to use the predictor, the on-device model must first be loaded. If you followed the Optional step above and included the ai.fritz:vision-style-painting-models dependency, you can get a predictor to use immediately:

Note

Use one of the included on-device models for the style you’d like to use:

// For included on-device models
PaintingStyles.BICENTENNIAL_PRINT
PaintingStyles.FEMMES
PaintingStyles.HEAD_OF_CLOWN
PaintingStyles.HORSES_ON_SEASHORE
PaintingStyles.KALEIDOSCOPE
PaintingStyles.PINK_BLUE_RHOMBUS
PaintingStyles.POPPY_FIELD
PaintingStyles.RITMO_PLASTICO
PaintingStyles.STARRY_NIGHT
PaintingStyles.THE_SCREAM
PaintingStyles.THE_TRAIL
FritzOnDeviceModel styleOnDeviceModel = PaintingStyles.STARRY_NIGHT;
FritzVisionStylePredictor predictor = FritzVision.StyleTransfer.getPredictor(styleOnDeviceModel);

If you did not include the on-device model with your app, you’ll have to load the model before you can get a predictor. To do that, you’ll use StyleManagedModel and call FritzVision.StyleTransfer.loadPredictor to start the model download.

Note

Use one of the managed models for the style you’d like to use:

// For models to download
PaintingManagedModels.BICENTENNIAL_PRINT_MANAGED_MODEL
PaintingManagedModels.FEMMES_MANAGED_MODEL
PaintingManagedModels.HEAD_OF_CLOWN_MANAGED_MODEL
PaintingManagedModels.HORSES_ON_SEASHORE_MANAGED_MODEL
PaintingManagedModels.KALEIDOSCOPE_MANAGED_MODEL
PaintingManagedModels.PINK_BLUE_RHOMBUS_MANAGED_MODEL
PaintingManagedModels.POPPY_FIELD_MANAGED_MODEL
PaintingManagedModels.RITMO_PLASTICO_MANAGED_MODEL
PaintingManagedModels.STARRY_NIGHT_MANAGED_MODEL
PaintingManagedModels.THE_SCREAM_MANAGED_MODEL
PaintingManagedModels.THE_TRAIL_MANAGED_MODEL
FritzVisionStylePredictor predictor;

FritzManagedModel managedModel = PaintingManagedModels.STARRY_NIGHT_MANAGED_MODEL;
FritzVision.StyleTransfer.loadPredictor(managedModel, new PredictorStatusListener<FritzVisionStylePredictor>() {
    @Override
    public void onPredictorReady(FritzVisionStylePredictor stylePredictor) {
        Log.d(TAG, "Style Transfer predictor is ready");
        predictor = stylePredictor;
    }
});

3. Create a FritzVisionImage from an image or a video stream

To create a FritzVisionImage from a Bitmap:

FritzVisionImage visionImage = FritzVisionImage.fromBitmap(bitmap);
var visionImage = FritzVisionImage.fromBitmap(bitmap)

To create a FritzVisionImage from a media.Image object when capturing the result from a camera, first determine the orientation of the image. This will rotate the image to account for device rotation and the orientation of the camera sensor.

// Get the system service for the camera manager
final CameraManager manager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);

// Gets the first camera id
String cameraId = manager.getCameraIdList().get(0);

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
ImageRotation imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId);
// Get the system service for the camera manager
val manager = getSystemService(Context.CAMERA_SERVICE) as CameraManager

// Gets the first camera id
var cameraId = manager.getCameraIdList().get(0)

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
var imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId)

Finally, create the FritzVisionImage object with the rotation

FritzVisionImage visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);
val visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);

4. Run prediction on FritzVisionImage

Apply a style to your FritzVisionImage to get a FritzVisionStyleResult object.

FritzVisionStyleResult styleResult = stylePredictor.predict(visionImage);

The predict method returns back a FritzVisionStyleResult object that contains the following methods:

FritzVisionSegmenFritzVisionStyleResulttationResult methods
Method Description
Bitmap toBitmap() Get the stylized image as a bitmap with the same dimensions as the model input resolution.
Bitmap toBitmap(Size scaleToSize) Get the stylized image as a bitmap with the same dimensions as the model input resolution.
void drawToCanvas(Canvas canvas) Draw the styled image to the canvas.
void drawToCanvas(Canvas canvas, Size canvasSize) Draw the styled image and scale it up to the specified size.

5. Access the Style Result

FritzVisionStyleResult contains several convenience methods to help draw the image.

Drawing the styled image to a canvas .. code-block:: java

// Draw the result to the canvas (same size as the input image) styleResult.drawToCanvas(canvas);

Accessing the bitmap

// Acccess the styled bitmap
Bitmap styledBitmap = styleResult.toBitmap();

Access a scaled bitmap

// Target canvas size
Size targetSize = new Size(2048, 2048);

Bitmap bitmap = styleResult.toBitmap(targetSize);

Advanced Options

Configuring the Predictor

You can configure the predictor with FritzVisionStylePredictorOptions to return specific results that match the options given:

FritzVisionStylePredictorOptions methods
Option Default Description
confidenceThreshold .3f Return labels above the confidence threshold
targetClasses Defined by the segmentation model The set of classes the model will look for

In order to change model performance for different devices, you may also expose the underlying TensorFlow Lite Interpreter options.

FritzVisionPredictorOptions methods
Option Default Description
useGPU false Return labels above the confidence threshold. Please note, this is an experimental option and should not be used in production apps.
useNNAPI false Uses the NNAPI for running model inference. Please note, this is an experimental option and should not be used in production apps.
numThreads <number of available threads> For CPU Only, run model inference using the specified number of threads

For more details, please visit the official TensorFlow Lite website.

Example:

Crop and scale option

By setting FritzVisionCropAndScale on FritzVisionStylePredictorOptions, you can control preprocessing on the image.

There are 2 crop and scale options available:

  • FritzVisionCropAndScale.SCALE_TO_FIT (default) - Resizes the original image to fit the size of the model input while maintaining the aspect ratio. After inference, the output is scaled back to the original aspect ratio. (e.g for a model input size of 384x384, scale a 600x400 image down to fit the model size. After the model runs inference, scale the output results to match the original 600x400 aspect ratio).
  • FritzVisionCropAndScale.CENTER_CROP - Run prediction on a center cropped section of the original image passed into the predict method. (e.g if the original image is 600x400, the predictor will run on a 400x400 image)

How to customize

If you want to train and use your own custom style transfer model, follow the open source training template on Google Colab.

For a full tutorial on training your custom model, take a look at this blog post on Heartbeat.

After you’ve finished training your model, follow these steps to add it to your Android app.

  1. Add your optimized TensorFlow Lite model (.tflite) to your app’s assets folder.
  2. Upload your custom model to Fritz’s webapp.
  3. Use the model id from Step 2 and initialize a predictor to use in your app.
FritzOnDeviceModel onDeviceModel = new FritzOnDeviceModel("file:///android_asset/custom_style_transfer_graph.tflite", "<Your model id from step 2>", 1);
predictor = FritzVision.StyleTransfer.getPredictor(onDeviceModel);
  1. Use your predictor the same way as described above with predictor.predict(visionImage).