Android

Note

If you haven’t set up the SDK yet, make sure to go through those directions first. You’ll need to add the Core library to the app before using the specific feature API or custom model. Follow iOS setup or Android setup directions.

Fritz provides an Android API that you can use to transform images or live video into beautiful works of art. You can choose to stylize an image from the photo gallery or transform live video captured by the device’s camera. Follow these simple instructions in order to bring style transfer to your app in no time.

Use Painting Styles

Follow these directions in order to apply one of the prebuilt styles to your app.

1. Add the dependencies via Gradle

Add our repository in order to download the Vision API:

repositories {
    maven { url "https://raw.github.com/fritzlabs/fritz-repository/master" }
}

Include the dependencies in app/build.gradle:

dependencies {
    implementation 'ai.fritz:core:3+'
    implementation 'ai.fritz:vision:3+'
}

(Optional include model in your app) To include Style Transfer model with your build, then you’ll need to add the dependency as shown below. Note: This includes the model with your app when you publish it to the play store and will increase your app size.

You’ll also need to specity aaptOptions in order to prevent compressing TensorFlow Lite models.

android {
  aaptOptions {
    noCompress "tflite"
  }
}

dependencies {
  implementation 'ai.fritz:vision-style-painting-models:3+'
}

Now you’re ready to transform images with the Style Transfer API.

2. Get a style predictor

In order to use the predictor, the on-device model must first be loaded. If you followed the Optional step above and included the ai.fritz:vision-style-painting-models dependency, you can get a predictor to use immediately:

Note

Use one of the included on-device models for the style you’d like to use:

// For included on-device models
PaintingStyles.BICENTENNIAL_PRINT
PaintingStyles.FEMMES
PaintingStyles.HEAD_OF_CLOWN
PaintingStyles.HORSES_ON_SEASHORE
PaintingStyles.KALEIDOSCOPE
PaintingStyles.PINK_BLUE_RHOMBUS
PaintingStyles.POPPY_FIELD
PaintingStyles.RITMO_PLASTICO
PaintingStyles.STARRY_NIGHT
PaintingStyles.THE_SCREAM
PaintingStyles.THE_TRAIL
import ai.fritz.ai.fritz.fritzvisionstylepaintings.PaintingStyles;
import ai.fritz.vision.styletransfer.FritzVisionStylePredictor;
import ai.fritz.core.FritzOnDeviceModel;
// ...

FritzOnDeviceModel styleOnDeviceModel = PaintingStyles.STARRY_NIGHT;
FritzVisionStylePredictor predictor = FritzVision.StyleTransfer.getPredictor(styleOnDeviceModel);

If you did not include the on-device model with your app, you’ll have to load the model before you can get a predictor. To do that, you’ll use StyleManagedModel and call FritzVision.StyleTransfer.loadPredictor to start the model download.

Note

Use one of the managed models for the style you’d like to use:

// For models to download
PaintingManagedModels.BICENTENNIAL_PRINT_MANAGED_MODEL
PaintingManagedModels.FEMMES_MANAGED_MODEL
PaintingManagedModels.HEAD_OF_CLOWN_MANAGED_MODEL
PaintingManagedModels.HORSES_ON_SEASHORE_MANAGED_MODEL
PaintingManagedModels.KALEIDOSCOPE_MANAGED_MODEL
PaintingManagedModels.PINK_BLUE_RHOMBUS_MANAGED_MODEL
PaintingManagedModels.POPPY_FIELD_MANAGED_MODEL
PaintingManagedModels.RITMO_PLASTICO_MANAGED_MODEL
PaintingManagedModels.STARRY_NIGHT_MANAGED_MODEL
PaintingManagedModels.THE_SCREAM_MANAGED_MODEL
PaintingManagedModels.THE_TRAIL_MANAGED_MODEL
import ai.fritz.vision.FritzVision;
import ai.fritz.vision.PredictorStatusListener;
import ai.fritz.vision.styletransfer.FritzVisionStylePredictor;
import ai.fritz.core.FritzManagedModel;
// ...

FritzVisionStylePredictor predictor;

FritzManagedModel managedModel = PaintingManagedModels.STARRY_NIGHT_MANAGED_MODEL;
FritzVision.StyleTransfer.loadPredictor(managedModel, new PredictorStatusListener<FritzVisionStylePredictor>() {
    @Override
    public void onPredictorReady(FritzVisionStylePredictor stylePredictor) {
        Log.d(TAG, "Style Transfer predictor is ready");
        predictor = stylePredictor;
    }
});

3. Create a FritzVisionImage from an image or a video stream

To create a FritzVisionImage from a Bitmap:

FritzVisionImage visionImage = FritzVisionImage.fromBitmap(bitmap);
var visionImage = FritzVisionImage.fromBitmap(bitmap)

To create a FritzVisionImage from a media.Image object when capturing the result from a camera, first determine the orientation of the image. This will rotate the image to account for device rotation and the orientation of the camera sensor.

// Get the system service for the camera manager
final CameraManager manager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);

// Gets the first camera id
String cameraId = manager.getCameraIdList().get(0);

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
int imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId);
// Get the system service for the camera manager
val manager = getSystemService(Context.CAMERA_SERVICE) as CameraManager

// Gets the first camera id
var cameraId = manager.getCameraIdList().get(0)

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
var imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId)

Finally, create the FritzVisionImage object with the rotation

FritzVisionImage visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);
val visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);

4. Run prediction on FritzVisionImage

Apply a style to your FritzVisionImage to get a FritzVisionStyleResult object.

FritzVisionStyleResult styleResult = stylePredictor.predict(visionImage);

The predict method returns back a FritzVisionStyleResult object that contains the following methods:

Method Description
FritzVisionImage getOriginalImage() Get the original image passed into the predict method.
Bitmap toBitmap() Get the stylized image as a bitmap with the same dimensions as the model input resolution.
Bitmap toBitmap(Size scaleToSize) Get the stylized image as a bitmap with the specified dimensions. (Note: This will add some processing time to resize the output)
FritzVisionImage getStyledImage() Get the styled image after it’s been passed through the model.
void drawVisionImage(Canvas canvas) Draws the original image passed in to the predict method.
void drawVisionImage(Canvas canvas, Size canvasSize) Draws the original image scaled up to the target canvas size. E.g Your original image was 512x512 but you’d like to stretch it to fit a canvas size of 1024x1024.
void drawToCanvas(Canvas canvas) Draw the styled image to the canvas.
void drawToCanvas(Canvas canvas, Size canvasSize) Draw the styled image and scale it up to the specified size.

5. Access the Style Result

FritzVisionStyleResult contains several convenience methods to help draw the image.

Drawing the styled image as the original size

Using a canvas to display the result:

// Draw the result to the canvas (same size as the input image)
styleResult.drawToCanvas(canvas);

Using an ImageView to display the result:

// Set the Bitmap for the image view
imageView.setBitmap(styleResult.getResultBitmap());

Drawing the styled image for a target size

Using a canvas to display the result:

// Target canvas size
Size targetSize = new Size(2048, 2048);

// Draw the result and scale the image to a specific size (while maintaining the aspect ratio)
styleResult.drawToCanvas(canvas, targetSize);

Using an ImageView to display the result:

// Target canvas size
Size targetSize = new Size(2048, 2048);

Bitmap bitmap = styleResult.getResultBitmap(targetSize);

// Set the scaled bitmap
imageView.setBitmap(scaledBitmap);

Accessing the bitmap directly

// Access the original bitmap
FritzVisionImage originalVisionImage = styleResult.getOriginalImage();
Bitmap originalBitmap = originalVisionImage.getBitmap();

// Acccess the styled bitmap
Bitmap styledBitmap = styleResult.getResultBitmap();

Configure Prediction

Before running style transfer, you can configure the prediction with a FritzVisionStylePredictorOptions object.

Settings
cropAndScaleOption

FritzVisionCropAndScale.SCALE_TO_FIT (default)

Crop and Scale option for how to resize and crop the image for the model

numThreads

1 (default)

The number of threads to use to process the result.

resolution

Size(width: 300, height: 400) (default)

The size of the input and output image.

Crop and scale option

By setting FritzVisionCropAndScale on FritzVisionStylePredictorOptions, you can control preprocessing on the image.

There are 2 crop and scale options available:

  • FritzVisionCropAndScale.SCALE_TO_FIT (default) - Resizes the original image to fit the size of the model input while maintaining the aspect ratio. After inference, the output is scaled back to the original aspect ratio. (e.g for a model input size of 384x384, scale a 600x400 image down to fit the model size. After the model runs inference, scale the output results to match the original 600x400 aspect ratio).
  • FritzVisionCropAndScale.CENTER_CROP - Run prediction on a center cropped section of the original image passed into the predict method. (e.g if the original image is 600x400, the predictor will run on a 400x400 image)

Example usage:

FritzVisionStylePredictorOptions options = new FritzVisionStylePredictorOptions.Builder()
  .cropAndScaleOption(FritzVisionCropAndScale.SCALE_TO_FIT)
  .build()
predictor = FritzVision.StyleTransfer.getPredictor(onDeviceModel, options);

Num Threads

Set the number of threads to use for model inference.

Example usage:

FritzVisionStylePredictorOptions options = new FritzVisionStylePredictorOptions.Builder()
  .numThreads(4)
  .build()
predictor = FritzVision.StyleTransfer.getPredictor(onDeviceModel, options);

Resolution

Set the desired resolution of the input and output image.

Example usage:

FritzVisionStylePredictorOptions options = new FritzVisionStylePredictorOptions.Builder()
  .resolution(new Size(480, 640))
  .build()
predictor = FritzVision.StyleTransfer.getPredictor(onDeviceModel, options);

Note

The resolution option must match the input size of your .tflite model. Fritz-trained painting models require the default resolution of 300 x 400 pixels. If you train your own model, the resolution must be set to whatever value you specified for --image-size when converting your model to TensorFlow Lite.


How to customize

If you want to train and use your own custom style transfer model, follow the open source training template on Google Colab.

For a full tutorial on training your custom model, take a look at this blog post on Heartbeat.

After you’ve finished training your model, follow these steps to add it to your Android app.

  1. Add your optimized TensorFlow Lite model (.tflite) to your app’s assets folder.
  2. Upload your custom model to Fritz’s webapp.
  3. Use the model id from Step 2 and initialize a predictor to use in your app.
FritzOnDeviceModel onDeviceModel = new FritzOnDeviceModel("file:///android_asset/custom_style_transfer_graph.tflite", "<Your model id from step 2>", 1);
predictor = FritzVision.StyleTransfer.getPredictor(onDeviceModel);
  1. Use your predictor the same way as described above with predictor.predict(visionImage).