Android Video Processing

Note

If you haven’t set up the SDK yet, make sure to go through those directions first. You’ll need to add the Core library to the app before using the specific feature API or custom model. Follow iOS setup or Android setup directions.

You can run predictions on an entire video, extract individual frames, and export processed video with FritzVisionVideo. Use FritzVisionImageFilter objects with FritzVisionVideo to run model predictions on every frame of a video.

1. Create FritzVisionVideo

Set up your FritzVisionVideo object to best suit your use case.

Set the source video

Specify path to video file

Load a video file with an absolute path or a URI.

FritzVisionVideo visionVideo = new FritzVisionVideo("file:///path/to/video");
// or
URI uri = URI.create("file:///path/to/video");
FritzVisionVideo visionVideo = new FritzVisionVideo(uri);

Initialize with filters

AI-powered effects are applied to videos via filters. A FritzVisionVideo can be initialized with a list of FritzVisionImageFilter objects that make model predictions and apply results to video frames.

Filters can be mixed and matched to customize the experience. If no filters are specified, FritzVisionVideo will be created without filters.

FritzOnDeviceModel starryNightModel = FritzVisionModels.getPaintingStyleModels().getStarryNight();
SegmentationOnDeviceModel peopleModel = FritzVisionModels.getPeopleSegmentationOnDeviceModel(ModelVariant.FAST);

FritzVisionStylePredictor stylePredictor = FritzVision.StyleTransfer.getPredictor(starryNightModel);
FritzVisionSegmentationPredictor peoplePredictor = FritzVision.ImageSegmentation.getPredictor(peopleModel);

FritzVisionImageFilter[] filters = new FritzVisionImageFilter[]{
    new StylizeImageCompoundFilter(stylePredictor),
    new MaskCutOutFilter(peoplePredictor, MaskClass.PERSON)
};

FritzVisionVideo visionVideo = new FritzVisionVideo("file:///path/to/video", filters);

2. Using FritzVisionImageFilter

By using FritzVisionVideo, you can transform frames and overlay images on top of them to visualize model results and create unique experiences and effects. This is accomplished through the use of FritzVisionImageFilter objects. You can even use several of these filter objects at once. Fritz includes pre-built filters for common tasks like drawing bounding boxes around objects or applying artistic styles, and it’s easy to make your own.

Pre-built filters

Get started with video processing by using the included pre-built filters.

Draw boxes around every detected object in your video.

FritzOnDeviceModel objectModel = FritzVisionModels.getObjectDetectionOnDeviceModel();
FritzVisionObjectPredictor objectPredictor = FritzVision.ObjectDetection.getPredictor(objectModel);

FritzVisionImageFilter filter = new DrawBoxesCompoundFilter(objectPredictor)
FritzVisionVideo visionVideo = new FritzVisionVideo("file:///path/to/video", filter)

Draw human pose skeletons on top of detected people in your video.

PoseOnDeviceModel poseModel = FritzVisionModels.getHumanPoseEstimationOnDeviceModel(ModelVariant.FAST);
FritzVisionPosePredictor posePredictor = FritzVision.PoseEstimation.getPredictor(poseModel);

FritzVisionImageFilter filter = DrawSkeletonCompoundFilter(poseModel);
FritzVisionVideo visionVideo = new FritzVisionVideo("file:///path/to/video", filter);

Stylize each frame of your video with unique effects.

FritzOnDeviceModel starryNightModel = FritzVisionModels.getPaintingStyleModels().getStarryNight();
FritzVisionStylePredictor stylePredictor = FritzVision.StyleTransfer.getPredictor(styleModel);

FritzVisionImageFilter filter = new StylizeImageCompoundFilter(stylePredictor);
FritzVisionVideo visionVideo = new FritzVisionVideo("file:///path/to/video", filter);

Blend the color of detected hair in your video with another color.

SegmentationOnDeviceModel hairModel = FritzVisionModels.getHairSegmentationOnDeviceModel(ModelVariant.FAST);
FritzVisionSegmentationPredictor hairPredictor = FritzVision.ImageSegmentation.getPredictor(hairModel);

FritzVisionImageFilter filter = new MaskBlendCompoundFilter(hairPredictor, MaskClass.HAIR, BlendMode.SOFT_LIGHT);
FritzVisionVideo visionVideo = new FritzVisionVideo("file:///path/to/video", filter);

Create an overlay mask in your video.

SegmentationOnDeviceModel hairModel = FritzVisionModels.getHairSegmentationOnDeviceModel(ModelVariant.FAST);
FritzVisionSegmentationPredictor hairPredictor = FritzVision.ImageSegmentation.getPredictor(hairModel);

FritzVisionImageFilter filter = new MaskOverlayFilter(hairPredictor, MaskClass.HAIR);
FritzVisionVideo visionVideo = FritzVisionVideo("file:///path/to/video", filter)

Cut out all a specific mask and make the background transparent.

SegmentationOnDeviceModel peopleModel = FritzVisionModels.getPeopleSegmentationOnDeviceModel(ModelVariant.FAST);
FritzVisionSegmentationPredictor peoplePredictor = FritzVision.ImageSegmentation.getPredictor(peopleModel);

FritzVisionImageFilter filter = new MaskCutOutOverlayFilter(peoplePredictor, MaskClass.PERSON);
FritzVisionVideo visionVideo = FritzVisionVideo("file:///path/to/video", filter)

Making your own filter

You can create your own filters by extending the FritzVisionImageFilter class and seamlessly integrate them with FritzVisionVideo.

FritzVisionImageFilter class overview

To create your own custom FritzVisionImageFilter, you’ll need to implement the processImage(FritzVisionImage image) method. This method returns a FritzVisionImage, which allows you to easily propogate successful predictions and any errors. You can denote how filtered frames are composited with original images by setting a filter’s FilterCompositionMode.

Implementing FritzVisionImageFilter

After making your custom filter, you can use it just like the pre-built filters and include it when initializing FritzVisionVideo.

public class MyCustomFilter extends FritzVisionImageFilter {

  @Override
  public FilterCompositionMode getCompositionMode() {
      return FilterCompositionMode.COMPOUND_WITH_PREVIOUS_OUTPUT;
  }

  public FritzVisionImage processImage(FritzVisionImage visionImage) {
    // Your code here
  }
}

Note

Filters can be configured to be COMPOUND_WITH_PREVIOUS_OUTPUT or OVERLAY_ON_ORIGINAL_IMAGE the source image.

OVERLAY_ON_ORIGINAL_IMAGE

An overlaying filter will use the original image as input into the predictor. Though filter results may be displayed as a layer on top of this original image, the input image itself is not modified. If two overlaying filters are used, the second filter will run using the original input image only and would not see any overlays from the first filter. For example, if the first overylay filter drew bounding boxes around objects, a second filter would receive an input image with those boxes hidden so that it would not impact the second filter’s model predictions. Generally, overlaying is best used with Image Segmentation as you can produce a masked image that will not cover the entire input image.

Overlaying filters

Overlaying with Hair Blend and People Mask.

COMPOUND_WITH_PREVIOUS_OUTPUT

A compounding filter will flatten the results from a previous filter and use this flattened image as the input to any subsequent filters. In the context of using multiple filters, the result of the previous filter will be used by the next filter. For example, if your first filter uses uses Style Transfer and your second filter uses Object Detection, the image being used by the second filter to make a prediction will be the stylized result of first. Keep filter order in mind, otherwise the accuracy of your predictions may be affected.

Compounding filters

Compounding with Pose Estimation and Object Detection.

3. Using FritzVisionVideo

Interact with your video in useful ways.

FritzVisionVideo can process a specific subset of frames to help fine-tune the experience and fulfill your performance requirements. Process frames in a larger interval to reduce total processing time, start at a specific frame, set a frame target, or do any combination of the three.

Extract specific frames

Getting a single frame

Extract a single frame with the getFrames method.

// Only fetch the 10th frame of the video.
FrameProcessingOptions params = new FrameProcessingOptions();
options.numFrames = 1;
options.startingFrameOffset = 10;

visionVideo.getFrames(options, new FritzVisionVideo.FrameProgressCallback() {
    @Override
    public void onProgress(FritzVisionImage response) {
        // Do something with each frame with the filters applied.
    }

    @Override
    public void onComplete() {
        // Do something once all frames have been processed.
    }
}))

Getting multiple frames

// Fetch frames 10 through 15.
FrameProcessingOptions params = new FrameProcessingOptions();
options.frameInterval = 1;
options.numFrames = 5;
options.startingFrameOffset = 10;

visionVideo.getFrames(options, new FritzVisionVideo.FrameProgressCallback() {
    @Override
    public void onProgress(FritzVisionImage response) {
        // Do something with each frame with the filters applied.
    }

    @Override
    public void onComplete() {
        // Do something once all frames have been processed.
    }
}))

Options for extracting frames:

FrameProcessingOptions
Option Default Description
frameInterval 1 The interval to process frames. In other words, process every n-th frame.
startingFrameOffset 0 The frame to start processing at.
numFrames # Frames in the video The number of frames to process.

Export video

Save your processed video to the Camera Roll.

Export the video frames with filters applied

File exportFile = File.createFile("exportedVideo.mp4");

fritzVideo.export(exportFile.getAbsolutePath(), new FritzVisionVideo.ExportProgressCallback() {
    @Override
    public void onProgress(Float response) {
        // Show updated progress.
    }

    @Override
    public void onComplete() {
        // when the file has completed
    }
});

Export the video frames with filters applied and audio

Note

Exporting the audio with the video will increase processing time. Currently audio exporting will only work when exporting the full video.

final File exportFile = File.createFile("exportedVideo.mp4");

// Export the video with audio and skip every other frame
// to improve the speed.
ExportVideoOptions options = new ExportVideoOptions();
options.frameInterval = 2;
options.copyAudio = true;

fritzVideo.export(exportFile.getAbsolutePath(), options, new FritzVisionVideo.ExportProgressCallback() {
    @Override
    public void onProgress(Float response) {
        // Show updated progress.
    }

    @Override
    public void onComplete() {
        // when the file has completed
    }
});

Options for exporting the video:

ExportVideoOptions
Option Default Description
frameInterval 1 The interval to process frames. In other words, process every n-th frame.
startingFrameOffset 0 The frame to start processing at.
numFrames # Frames in the video The number of frames to process.
bitRate Estimated value The visual quality of the video. Bit rate will be automatically estimated based on the resolution and frame rate if a value is not set.
frameRateScale 1.0 The value to scale the frame rate by.
keyFrameInterval 0 The frequency that key frames are requested per second. An interval of 0 means every frame will be a key frame.
copyAudio false If audio should be exported. Increases processing time when enabled. Currently audio exporting is only supported with frameRateScale=1 and startingFrameOffset=0.

4. Display Processed Video

Use a VideoView to display your processed video.

For more details, visit the official Android documentation for VideoView.

VideoView videoView = findViewById(R.id.video_view);
videoView.setOnPreparedListener(new MediaPlayer.OnPreparedListener() {
    @Override
    public void onPrepared(MediaPlayer mediaPlayer) {
        mediaPlayer.setLooping(true);
        mediaPlayer.setVolume(1, 1);
        mediaPlayer.start();
    }
});

// After the video finishes

fritzVideo.export(exportFile.getAbsolutePath(), options, new FritzVisionVideo.ExportProgressCallback() {
    @Override
    public void onProgress(Float response) {
        // Show updated progress.
    }

    @Override
    public void onComplete() {
        // when the file has completed, add the video to the view to play it.
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                Uri uri = Uri.fromFile(exportFile);
                videoView.setVideoURI(uri);
            }
        });
    }
});

Additional Resources

Want more resources to get started? Check out the following: