Android

Fritz provides an Android API that you can use to transform images or live video into beautiful works of art. You can choose to stylize an image from the photo gallery or transform live video captured by the device’s camera. Follow these simple instructions in order to bring style transfer to your app in no time.

Note

If you haven’t set up the SDK yet, make sure to go through those directions first. You’ll need to add the Core library to the app before using the specific feature or custom model libraries.

Apply Painting Styles

Follow these directions in order to apply one of the prebuilt styles to your app.

1. Add FritzVisionStyleModel via Gradle

Assuming that you’ve already included the Core SDK and the fritz repository, you’ll also want to include the style transfer dependency.

In app/build.gradle:

dependencies {
    implementation 'ai.fritz:core:2.0.0'

    implementation 'ai.fritz:vision-style:2.0.0'
    implementation 'ai.fritz:vision-style-paintings:2.0.0'
}

2. Get a style predictor

First, get a predictor class for one of our 11 different styles.

// Define the stylePredictor
FritzVisionStylePredictor stylePredictor;

// Get the starry night predictor
// "this" refers to the calling Context (Application, Activity, etc)
stylePredictor = FritzVisionStyleTransfer.getPredictor(PaintingStyles.STARRY_NIGHT)

3. Create a FritzVisionImage from an image or a video stream

To create a FritzVisionImage from a Bitmap:

FritzVisionImage visionImage = FritzVisionImage.fromBitmap(bitmap);

To create a FritzVisionImage from a media.Image object when capturing the result from a camera, first determine the orientation of the image. This will rotate the image to account for device rotation and the orientation of the camera sensor.

// Get the system service for the camera manager
final CameraManager manager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);

// Gets the first camera id
String cameraId = manager.getCameraIdList().get(0);

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
int imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId);

Finally, create the FritzVisionImage object with the rotation

FritzVisionImage visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);

4. Run prediction on FritzVisionImage

Apply a style to your FritzVisionImage to get a FritzVisionStyleResult object.

FritzVisionStyleResult styleResult = stylePredictor.predict(visionImage);

The predict method returns back a FritzVisionStyleResult object that contains the following methods:

Method Description
FritzVisionImage getOriginalImage() Get the original image passed into the predict method.
FritzVisionImage getStyledImage() Get the styled image after it’s been passed through the model.
void drawVisionImage(Canvas canvas) Draws the original image passed in to the predict method.
void drawVisionImage(Canvas canvas, Size canvasSize) Draws the original image scaled up to the target canvas size. E.g Your original image was 512x512 but you’d like to stretch it to fit a canvas size of 1024x1024.
void drawToCanvas(Canvas canvas) Draw the styled image to the canvas.
void drawToCanvas(Canvas canvas, Size canvasSize) Draw the styled image and scale it up to the specified size.

5. Access the Style Result

FritzVisionStyleResult contains several convenience methods to help draw the image.

Drawing the styled image as the original size

// Draw the result to the canvas (same size as the input image)
Canvas canvas = new Canvas();
styleResult.drawToCanvas(canvas);

Drawing the styled image on a target canvas size

Canvas canvas = new Canvas();

// Target canvas size
Size targetSize = new Size(2048, 2048);

// Draw the result and scale the image to a specific size (while maintaining the aspect ratio)
styleResult.drawToCanvas(canvas, targetSize);

Accessing the bitmap directly

// Access the original bitmap
FritzVisionImage originalVisionImage = styleResult.getOriginalImage();
Bitmap originalBitmap = originalVisionImage.getBitmap();

// Acccess the styled bitmap
FritzVisionImage resultVisionImage = styleResult.getStyledImage();
Bitmap styledBitmap = resultVisionImage.getBitmap();

Note

Speed vs Resolution

To achieve higher resolution on the stylized image, you can specify FritzVisionStylePredictorOptions with the image resolution set to HIGH, however, keep in mind, that higher resolution results in longer prediction times.

If you’d like to run style transfer on a video frame, set FritzStyleResolution to LOW or NORMAL.

  • To initialize the options when getting the style predictor.

    FritzVisionStylePredictorOptions options = new FritzVisionStylePredictorOptions.Builder()
            .imageResolution(FritzStyleResolution.HIGH)
            .build();
    
    predictor = FritzVisionStyleTransfer.getPredictor(ArtisticStyle.STARRY_NIGHT, options);
    
  • To change the options on an existing predictor.

    FritzVisionStylePredictorOptions options = new FritzVisionStylePredictorOptions.Builder()
            .imageResolution(FritzStyleResolution.HIGH)
            .build();
    predictor.setOptions(options);
    

Create a Custom Style Transfer Model

If you want to train and use your own custom style transfer model, follow our open source training template on Google Colab.

For a full tutorial on training your custom model, take a look at this blog post on Heartbeat.

After you’ve finished training your model, follow these steps to add it to your Android app.

  1. Add your optimized TensorFlow Mobile protobuf file (.pb) that contains your custom trained style transfer model to your app’s assets folder.
  2. Upload your custom model to Fritz’s webapp.
  3. In app/build.gradle, you only need to use the vision-style library (this library doesn’t include the 11 prebuilt styles, keeping your apk size small).
dependencies {
    implementation 'ai.fritz:core:2.0.0'
    implementation 'ai.fritz:vision-style:2.0.0'
}
  1. Use the model id from Step 2 and initialize a predictor to use in your app.
CustomModel customModel = new CustomModel("custom_style_transfer_graph.pb", "<Your model id from step 2>", 1);
predictor = FritzVisionStyleTransfer.getCustomPredictor(customModel);
  1. Use your predictor the same way as described above with predictor.predict(visionImage).