iOS Video Processing

Note

If you haven’t set up the SDK yet, make sure to go through those directions first. You’ll need to add the Core library to the app before using the specific feature API or custom model. Follow iOS setup or Android setup directions.

You can run predictions on an entire video, extract individual frames, and export processed video with FritzVisionVideo. Use FritzVisionImageFilter objects with FritzVisionVideo to run model predictions on every frame of a video.

1. Create FritzVisionVideo

Set up your FritzVisionVideo object to best suit your use case.

Set the source video

Specify path to video file

Target a video file locally or in the cloud.

import Fritz

let fritzVideo = FritzVisionVideo(path: "file:///path/to/video")
// OR
// let fritzVideo = FritzVisionVideo(url: videoUrl)

Use an existing video

If you already have an AVPlayer object, you can start processing it with FritzVisionVideo.

import Fritz

let fritzVideo = FritzVisionVideo(player: myAVPlayer)

Initialize with filters

AI-powered effects are applied to videos via filters. A FritzVisionVideo can be initialized with a list of FritzVisionImageFilter objects that make model predictions and apply results to video frames.

Filters can be mixed and matched to customize the experience. If no filters are specified, FritzVisionVideo will be created without filters.

import Fritz

let hairModel = FritzVisionHairSegmentationModelFast()
let objectModel = FritzVisionObjectModelFast()
let filters: [FritzVisionImageFilter] = [
  FritzVisionBlendHairCompoundFilter(model: hairModel),
  FritzVisionDrawBoxesCompoundFilter(model: objectModel)
]

let fritzVideo = FritzVisionVideo(player: myAVPlayer, applyingFilters: filters)

2. Using FritzVisionImageFilter

By using FritzVisionVideo, you can transform frames and overlay images on top of them to visualize model results and create unique experiences and effects. This is accomplished through the use of FritzVisionImageFilter objects. You can even use several of these filter objects at once. Fritz includes pre-built filters for common tasks like drawing bounding boxes around objects or applying artistic styles, and it’s easy to make your own.

Pre-built filters

Get started with video processing by using the included pre-built filters.

Draw boxes around every detected object in your video.

import Fritz

let objectModel = FritzVisionObjectModelFast()
let filter = FritzVisionDrawBoxesCompoundFilter(model: objectModel)

let fritzVideo = FritzVisionVideo(url: url, withFilter: filter)

Draw human pose skeletons on top of detected people in your video.

import Fritz

let poseModel = FritzVisionHumanPoseModelFast()
let filter = FritzVisionDrawSkeletonCompoundFilter(model: poseModel)

let fritzVideo = FritzVisionVideo(url: url, withFilter: filter)

Stylize each frame of your video with unique effects.

import Fritz

let styleModel = PatternModels.Style.snowflake.build()
let filter = FritzVisionStylizeImageCompoundFilter(model: styleModel)

let fritzVideo = FritzVisionVideo(url: url, withFilter: filter)

Blend the color of detected hair in your video with another color.

import Fritz

let hairModel = FritzVisionHairSegmentationModelFast()
let filter = FritzVisionBlendHairCompoundFilter(model: hairModel)

let fritzVideo = FritzVisionVideo(url: url, withFilter: filter)

Create a mask over detected hair in your video.

import Fritz

let hairModel = FritzVisionHairSegmentationModelFast()
let filter = FritzVisionMaskHairOverlayFilter(model: hairModel)

let fritzVideo = FritzVisionVideo(url: url, withFilter: filter)

Create a mask over all detected people in your video.

import Fritz

let peopleModel = FritzVisionPeopleSegmentationModelFast()
let filter = FritzVisionMaskPeopleOverlayFilter(model: peopleModel)

let fritzVideo = FritzVisionVideo(url: url, withFilter: filter)

Cut out all detected people in a frame and make the background transparent.

import Fritz

let peopleModel = FritzVisionPeopleSegmentationModelFast()
let filter = FritzVisionCutOutPeopleOverlayFilter(model: peopleModel)

let fritzVideo = FritzVisionVideo(url: url, withFilter: filter)

Making your own filter

You can create your own filters by implementing the FritzVisionImageFilter protocol and seamlessly integrate them with FritzVisionVideo.

FritzVisionImageFilter protocol overview

To create your own custom FritzVisionImageFilter, you’ll need to implement the process() method. This method returns a Result, which allows you to easily propogate successful predictions and any errors. You can denote how filtered frames are composited with original images by setting a filter’s FilterCompositionMode.

public protocol FritzVisionImageFilter {

  typealias FritzVisionFilterResult = Result<FritzVisionImage, Error>

  /// How predictors are chosen by the filter.
  var compositionMode: FilterCompositionMode { get }

  /// Processes the image with a prediction.
  ///
  /// - Parameters:
  ///   - image: the image to process
  /// - Returns: the result of the prediction
  func process(_ image: FritzVisionImage) -> FritzVisionFilterResult
}

Implementing FritzVisionImageFilter

After making your custom filter, you can use it just like the pre-built filters and include it when initializing FritzVisionVideo.

import Fritz

public class MyCustomFilter: FritzVisionImageFilter {

  public let compositionMode = FilterCompositionMode.overlayOnOriginalImage
  // OR
  // public let compositionMode = FilterCompositionMode.compoundWithPreviousOutput

  public func process(_ image: FritzVisionImage) -> FritzVisionFilterResult {
    // Your code here
  }
}

Note

Filters can be configured to be overlaying or compounding the source image.

Overlaying

An overlaying filter will use the original image as input into the predictor. Though filter results may be displayed as a layer on top of this original image, the input image itself is not modified. If two overlaying filters are used, the second filter will run using the original input image only and would not see any overlays from the first filter. For example, if the first overylay filter drew bounding boxes around objects, a second filter would receive an input image with those boxes hidden so that it would not impact the second filter’s model predictions. Generally, overlaying is best used with Image Segmentation as you can produce a masked image that will not cover the entire input image.

Overlaying filters

Overlaying with Hair Blend and People Mask.

Compounding

A compounding filter will flatten the results from a previous filter and use this flattened image as input to any models. In the context of using multiple filters, the result of the previous filter will be used by the next filter. For example, if your first filter uses uses Style Transfer and your second filter uses Object Detection, the image being used by the second filter to make a prediction will be the stylized result of first. Keep filter order in mind, otherwise the accuracy of your predictions may be affected.

Compounding filters

Compounding with Pose Estimation and Object Detection.

3. Using FritzVisionVideo

Interact with your video in useful ways.

Extract frames

Getting a single frame

Make a synchronous call to extract a frame. Useful when a single frame is needed, like in video thumbnail generation.

// Frame at time 0.6s with all filters applied
let frame = fritzVideo.frame(at: 0.6)

// Frame without any filters applied
let originalFrame = fritzVideo.frame(at: 0.6, processed: false)

Getting multiple frames

Make an asynchronous call to extract several frames.

// Frames at specified times with all filters applied
fritzVideo.frames(at: [0.6, 1, 1.8, 3]) { response in
  switch response {
  case .success(let image):
    // Handle frame retrieval
  case .failure(let error):
    // Handle error
}

Note

When extracting multiple frames from a video, make sure to call the .frames() method instead of calling .frame() in a loop. Multiple calls to .frame() will block the main thread, whereas .frames() prevents that by extracting frames asynchronously. Use a completion handler to handle successful extractions and errors as they propogate.

Stitch videos together

You can append additional video content to your source video.

import Fritz

let myAVAsset = myAVPlayer.currentItem!.asset
try? fritzVideo.stitch(with: myAVAsset)

Export video

Save your processed video to the Camera Roll.

let url = URL(fileURLWithPath: "file:///path/to/file.mov")
let exporter: AVAssetExportSession = fritzVideo.export(to: url, as: AVFileType.mov) { response in
  switch response {
  case .success(let outputUrl):
    // Handle export success
  case .failure(let error):
    // Handle error
}

// Track progress of the export
let currentProgress = exporter.progress

Note

When exporting a video, the output URL must include a file extension. This file extension should match your provided AVFileType. Otherwise, the video export will result in a failure.

Modify playback

You can change the framerate of your video for exporting and playback by scaling its native value. You are also able to loop the video during playback.

// Doubling the framerate of the video
fritzVideo.frameRateScale = 2.0

// Stop the video from looping
fritzVideo.loop = false

4. Display Processed Video

Use FritzVideoView to easily display your processed video.

// Setup the view
let fritzView = FritzVideoView()
fritzView.frame = view.bounds
fritzView.fritzVideo = videoPlayer
view.bringSubviewToFront(fritzView)

// Play the video
fritzView.play()

Additional Resources

Want more resources to get started? Check out the following: