Video Processing for Mobile Apps

Fritz Vision includes a comprehensive API to process pre-recorded video. Features include running predictions on every frame, extracting individual frames, and exporting video. Turn old experiences into new ones with machine learning augmentations.

Use Filters

Predictions are made on every frame using filters. Use our pre-built filters or implement your own. Mix and match them to customize your experience.

Pre-Recorded Video

Sometimes ML models just can't keep up with real-time video on every device. FritzVisionVideo makes it easy to process frames of pre-recorded videos and render outputs that can be smoothly played back at their original frame rate.

iOS

note

If you haven't set up the SDK yet, make sure to go through those directions first. You'll need to add the Core library to the app before using the specific feature API or custom model. Follow iOS setup or Android setup directions.

You can run predictions on an entire video, extract individual frames, and export processed video with FritzVisionVideo. Use FritzVisionImageFilter objects with FritzVisionVideo to run model predictions on every frame of a video.

1. Create FritzVisionVideo

Set up your FritzVisionVideo object to best suit your use case.

Set the source video

Specify path to video file

Target a video file locally or in the cloud.

import Fritz
let fritzVideo = FritzVisionVideo(path: "file:///path/to/video")
// OR
// let fritzVideo = FritzVisionVideo(url: videoUrl)
Use an existing video

If you already have an AVPlayer object, you can start processing it with FritzVisionVideo.

import Fritz
let fritzVideo = FritzVisionVideo(player: myAVPlayer)
Initialize with filters

AI-powered effects are applied to videos via filters. A FritzVisionVideo can be initialized with a list of FritzVisionImageFilter objects that make model predictions and apply results to video frames.

Filters can be mixed and matched to customize the experience. If no filters are specified, FritzVisionVideo will be created without filters.

import Fritz
let hairModel = FritzVisionHairSegmentationModelFast()
let objectModel = FritzVisionObjectModelFast()
let filters: [FritzVisionImageFilter] = [
FritzVisionBlendHairCompoundFilter(model: hairModel),
FritzVisionDrawBoxesCompoundFilter(model: objectModel)
]
let fritzVideo = FritzVisionVideo(player: myAVPlayer, applyingFilters: filters)

2. Using FritzVisionImageFilter

By using FritzVisionVideo, you can transform frames and overlay images on top of them to visualize model results and create unique experiences and effects. This is accomplished through the use of FritzVisionImageFilter objects. You can even use several of these filter objects at once. Fritz includes pre-built filters for common tasks like drawing bounding boxes around objects or applying artistic styles, and it's easy to make your own.

Pre-built filters

Get started with video processing by using the included pre-built filters.

Draw boxes around every detected object in your video.
import Fritz
let objectModel = FritzVisionObjectModelFast()
let filter = FritzVisionDrawBoxesCompoundFilter(model: objectModel)
let fritzVideo = FritzVisionVideo(url: url, withFilter: filter)

Making your own filter

You can create your own filters by implementing the FritzVisionImageFilter protocol and seamlessly integrate them with FritzVisionVideo.

FritzVisionImageFilter protocol overview

To create your own custom FritzVisionImageFilter, you'll need to implement the process() method. This method returns a Result, which allows you to easily propogate successful predictions and any errors. You can denote how filtered frames are composited with original images by setting a filter's FilterCompositionMode.

public protocol FritzVisionImageFilter {
typealias FritzVisionFilterResult = Result<FritzVisionImage, Error>
/// How predictors are chosen by the filter.
var compositionMode: FilterCompositionMode { get }
/// Processes the image with a prediction.
///
/// - Parameters:
/// - image: the image to process
/// - Returns: the result of the prediction
func process(_ image: FritzVisionImage) -> FritzVisionFilterResult
}

Implementing FritzVisionImageFilter

After making your custom filter, you can use it just like the pre-built filters and include it when initializing FritzVisionVideo.

import Fritz
public class MyCustomFilter: FritzVisionImageFilter {
public let compositionMode = FilterCompositionMode.overlayOnOriginalImage
// OR
// public let compositionMode = FilterCompositionMode.compoundWithPreviousOutput
public func process(_ image: FritzVisionImage) -> FritzVisionFilterResult {
// Your code here
}
}
note

Filters can be configured to be overlaying or compounding the source image.

Overlaying

An overlaying filter will use the original image as input into the predictor. Though filter results may be displayed as a layer on top of this original image, the input image itself is not modified. If two overlaying filters are used, the second filter will run using the original input image only and would not see any overlays from the first filter. For example, if the first overylay filter drew bounding boxes around objects, a second filter would receive an input image with those boxes hidden so that it would not impact the second filter's model predictions. Generally, overlaying is best used with Image Segmentation as you can produce a masked image that will not cover the entire input image.

Overlaying with Hair Blend and People Mask.

Overlaying filters

Compounding

A compounding filter will flatten the results from a previous filter and use this flattened image as input to any models. In the context of using multiple filters, the result of the previous filter will be used by the next filter. For example, if your first filter uses uses Style Transfer and your second filter uses Object Detection, the image being used by the second filter to make a prediction will be the stylized result of first. Keep filter order in mind, otherwise the accuracy of your predictions may be affected.

Compounding with Pose Estimation and Object Detection.

v filters

3. Using FritzVisionVideo

Interact with your video in useful ways.

Extract frames

Getting a single frame

Make a synchronous call to extract a frame. Useful when a single frame is needed, like in video thumbnail generation.

// Frame at time 0.6s with all filters applied
let frame = fritzVideo.frame(at: 0.6)
// Frame without any filters applied
let originalFrame = fritzVideo.frame(at: 0.6, processed: false)
Getting multiple frames

Make an asynchronous call to extract several frames.

// Frames at specified times with all filters applied
fritzVideo.frames(at: [0.6, 1, 1.8, 3]) { response in
switch response {
case .success(let image):
// Handle frame retrieval
case .failure(let error):
// Handle error
}
note

When extracting multiple frames from a video, make sure to call the .frames() method instead of calling .frame() in a loop. Multiple calls to .frame() will block the main thread, whereas .frames() prevents that by extracting frames asynchronously. Use a completion handler to handle successful extractions and errors as they propogate.

Stitch videos together

You can append additional video content to your source video.

import Fritz
let myAVAsset = myAVPlayer.currentItem!.asset
try? fritzVideo.stitch(with: myAVAsset)

Export video

Save your processed video to the Camera Roll.

let url = URL(fileURLWithPath: "file:///path/to/file.mov")
let exporter: AVAssetExportSession = fritzVideo.export(to: url, as: AVFileType.mov) { response in
switch response {
case .success(let outputUrl):
// Handle export success
case .failure(let error):
// Handle error
}
// Track progress of the export
let currentProgress = exporter.progress
note

When exporting a video, the output URL must include a file extension. This file extension should match your provided AVFileType. Otherwise, the video export will result in a failure.

Modify playback

You can change the framerate of your video for exporting and playback by scaling its native value. You are also able to loop the video during playback.

// Doubling the framerate of the video
fritzVideo.frameRateScale = 2.0
// Stop the video from looping
fritzVideo.loop = false

4. Display Processed Video

Use FritzVideoView to easily display your processed video.

// Setup the view
let fritzView = FritzVideoView()
fritzView.frame = view.bounds
fritzView.fritzVideo = videoPlayer
view.bringSubviewToFront(fritzView)
// Play the video
fritzView.play()

Additional Resources

Want more resources to get started? Check out the following:


Android

note

If you haven't set up the SDK yet, make sure to go through those directions first. You'll need to add the Core library to the app before using the specific feature API or custom model. Follow iOS setup or Android setup directions.

You can run predictions on an entire video, extract individual frames, and export processed video with FritzVisionVideo. Use FritzVisionImageFilter objects with FritzVisionVideo to run model predictions on every frame of a video.

1. Create a FritzVisionVideo

Set up your FritzVisionVideo object to best suit your use case.

Set the source video

Specify path to video file

Load a video file with an absolute path or a URI.

FritzVisionVideo visionVideo = new FritzVisionVideo("file:///path/to/video");
// or
URI uri = URI.create("file:///path/to/video");
FritzVisionVideo visionVideo = new FritzVisionVideo(uri);
Initialize with filters

AI-powered effects are applied to videos via filters. A FritzVisionVideo can be initialized with a list of FritzVisionImageFilter objects that make model predictions and apply results to video frames.

Filters can be mixed and matched to customize the experience. If no filters are specified, FritzVisionVideo will be created without filters.

FritzOnDeviceModel starryNightModel = FritzVisionModels.getPaintingStyleModels().getStarryNight();
SegmentationOnDeviceModel peopleModel = FritzVisionModels.getPeopleSegmentationOnDeviceModel(ModelVariant.FAST);
FritzVisionStylePredictor stylePredictor = FritzVision.StyleTransfer.getPredictor(starryNightModel);
FritzVisionSegmentationPredictor peoplePredictor = FritzVision.ImageSegmentation.getPredictor(peopleModel);
FritzVisionImageFilter[] filters = new FritzVisionImageFilter[]{
new StylizeImageCompoundFilter(stylePredictor),
new MaskCutOutFilter(peoplePredictor, MaskClass.PERSON)
};
FritzVisionVideo visionVideo = new FritzVisionVideo("file:///path/to/video", filters);

2. Using FritzVisionImageFilter

By using FritzVisionVideo, you can transform frames and overlay images on top of them to visualize model results and create unique experiences and effects. This is accomplished through the use of FritzVisionImageFilter objects. You can even use several of these filter objects at once. Fritz includes pre-built filters for common tasks like drawing bounding boxes around objects or applying artistic styles, and it's easy to make your own.

Pre-built filters

Get started with video processing by using the included pre-built filters.

Draw boxes around every detected object in your video.

FritzOnDeviceModel objectModel = FritzVisionModels.getObjectDetectionOnDeviceModel();
FritzVisionObjectPredictor objectPredictor = FritzVision.ObjectDetection.getPredictor(objectModel);
FritzVisionImageFilter filter = new DrawBoxesCompoundFilter(objectPredictor)
FritzVisionVideo visionVideo = new FritzVisionVideo("file:///path/to/video", filter)

Making your own filter

You can create your own filters by extending the FritzVisionImageFilter class and seamlessly integrate them with FritzVisionVideo.

FritzVisionImageFilter class overview

To create your own custom FritzVisionImageFilter, you'll need to implement the processImage(FritzVisionImage image) method. This method returns a FritzVisionImage, which allows you to easily propogate successful predictions and any errors. You can denote how filtered frames are composited with original images by setting a filter's FilterCompositionMode.

Implementing FritzVisionImageFilter

After making your custom filter, you can use it just like the pre-built filters and include it when initializing FritzVisionVideo.

public class MyCustomFilter extends FritzVisionImageFilter {
@Override
public FilterCompositionMode getCompositionMode() {
return FilterCompositionMode.COMPOUND_WITH_PREVIOUS_OUTPUT;
}
public FritzVisionImage processImage(FritzVisionImage visionImage) {
// Your code here
}
}
note

Filters can be configured to be COMPOUND_WITH_PREVIOUS_OUTPUT or OVERLAY_ON_ORIGINAL_IMAGE the source image.

OVERLAY_ON_ORIGINAL_IMAGE

An overlaying filter will use the original image as input into the predictor. Though filter results may be displayed as a layer on top of this original image, the input image itself is not modified. If two overlaying filters are used, the second filter will run using the original input image only and would not see any overlays from the first filter. For example, if the first overylay filter drew bounding boxes around objects, a second filter would receive an input image with those boxes hidden so that it would not impact the second filter's model predictions. Generally, overlaying is best used with Image Segmentation as you can produce a masked image that will not cover the entire input image.

Overlaying with Hair Blend and People Mask.

Overlaying filters

COMPOUND_WITH_PREVIOUS_OUTPUT

A compounding filter will flatten the results from a previous filter and use this flattened image as the input to any subsequent filters. In the context of using multiple filters, the result of the previous filter will be used by the next filter. For example, if your first filter uses uses Style Transfer and your second filter uses Object Detection, the image being used by the second filter to make a prediction will be the stylized result of first. Keep filter order in mind, otherwise the accuracy of your predictions may be affected.

Compounding with Pose Estimation and Object Detection.

Compounding filters

3. Using FritzVisionVideo

Interact with your video in useful ways.

FritzVisionVideo can process a specific subset of frames to help fine-tune the experience and fulfill your performance requirements. Process frames in a larger interval to reduce total processing time, start at a specific frame, set a frame target, or do any combination of the three.

Extract specific frames

Getting a single frame

Extract a single frame with the getFrames method.

// Only fetch the 10th frame of the video.
FrameProcessingOptions params = new FrameProcessingOptions();
options.numFrames = 1;
options.startingFrameOffset = 10;
visionVideo.getFrames(options, new FritzVisionVideo.FrameProgressCallback() {
@Override
public void onProgress(FritzVisionImage response) {
// Do something with each frame with the filters applied.
}
@Override
public void onComplete() {
// Do something once all frames have been processed.
}
}))
Getting multiple frames
// Fetch frames 10 through 15.
FrameProcessingOptions params = new FrameProcessingOptions();
options.frameInterval = 1;
options.numFrames = 5;
options.startingFrameOffset = 10;
visionVideo.getFrames(options, new FritzVisionVideo.FrameProgressCallback() {
@Override
public void onProgress(FritzVisionImage response) {
// Do something with each frame with the filters applied.
}
@Override
public void onComplete() {
// Do something once all frames have been processed.
}
}))
Options for extracting frames:

FrameProcessingOptions

OptionDefaultDescription
frameInterval1The interval to process frames. In other words, process every n-th frame.
startingFrameOffset0The frame to start processing at.
numFrames# Frames in the videoThe number of frames to process.

Export video

Save your processed video to the Camera Roll.

Export the video frames with filters applied
File exportFile = File.createFile("exportedVideo.mp4");
fritzVideo.export(exportFile.getAbsolutePath(), new FritzVisionVideo.ExportProgressCallback() {
@Override
public void onProgress(Float response) {
// Show updated progress.
}
@Override
public void onComplete() {
// when the file has completed
}
});
Export the video frames with filters applied and audio
note

Exporting the audio with the video will increase processing time. Currently audio exporting will only work when exporting the full video.

final File exportFile = File.createFile("exportedVideo.mp4");
// Export the video with audio and skip every other frame
// to improve the speed.
ExportVideoOptions options = new ExportVideoOptions();
options.frameInterval = 2;
options.copyAudio = true;
fritzVideo.export(exportFile.getAbsolutePath(), options, new FritzVisionVideo.ExportProgressCallback() {
@Override
public void onProgress(Float response) {
// Show updated progress.
}
@Override
public void onComplete() {
// when the file has completed
}
});
Options for exporting the video:

ExportVideoOptions

OptionDefaultDescription
frameInterval1The interval to process frames. In other words, process every n-th frame.
startingFrameOffset0The frame to start processing at.
numFrames# Frames in the videoThe number of frames to process.
bitRateEstimated valueThe visual quality of the video. Bit rate will be automatically estimated based on the resolution and frame rate if a value is not set.
frameRateScale1.0The value to scale the frame rate by.
keyFrameInterval0The frequency that key frames are requested per second. An interval of 0 means every frame will be a key frame.
copyAudiofalseIf audio should be exported. Increases processing time when enabled. Currently audio exporting is only supported with frameRateScale=1 and startingFrameOffset=0.

4. Display Processed Video

Use a VideoView to display your processed video.

For more details, visit the official Android documentation for VideoView.

VideoView videoView = findViewById(R.id.video_view);
videoView.setOnPreparedListener(new MediaPlayer.OnPreparedListener() {
@Override
public void onPrepared(MediaPlayer mediaPlayer) {
mediaPlayer.setLooping(true);
mediaPlayer.setVolume(1, 1);
mediaPlayer.start();
}
});
// After the video finishes
fritzVideo.export(exportFile.getAbsolutePath(), options, new FritzVisionVideo.ExportProgressCallback() {
@Override
public void onProgress(Float response) {
// Show updated progress.
}
@Override
public void onComplete() {
// when the file has completed, add the video to the view to play it.
runOnUiThread(new Runnable() {
@Override
public void run() {
Uri uri = Uri.fromFile(exportFile);
videoView.setVideoURI(uri);
}
});
}
});

Additional Resources

Want more resources to get started? Check out the following: