Image Labeling on iOS

Note

If you haven’t set up the SDK yet, make sure to go through those directions first. You’ll need to add the Core library to the app before using the specific feature API or custom model. Follow iOS setup or Android setup directions.

You can use the FritzVisionLabelModel to label the contents of images. Fritz provides a variety of options to configure predictions.

1. Build the FritzVisionLabelModel

To create the label model, you can either include the model in your bundle or download it over the air once the user installs your app.

Include the model in your application bundle

Add the model to your Podfile

Include Fritz/VisionLabelModel in your Podfile. This will include the model file in your app bundle.

pod 'Fritz/VisionLabelModel/Fast'

Make sure to install the recent addition.

pod install

Note

If you’ve built the app with just the core Fritz pod and add a new submodule for the model, you may encounter an error “Cannot invoke initializer for type”. To fix this, run a pod update and clean your XCode build to resolve the issue.

Define FritzVisionLabelModelFast

Define the instance of the FritzVisionLabelModelFast in your code. There should only be one instance that is reused for each prediction.

import Fritz

let labelModel = FritzVisionLabelModelFast()
@import Fritz;

FritzVisionLabelModelFast *labelModel = [FritzVisionLabelModelFastObjc model];

Note

Model initialization

It’s important to intialize one instance of the model so you are not loading the entire model into memory on each model execution. Usually this is a property on a ViewController. When loading the model in a ViewController, the following ways are recommended:

Lazy-load the model

By lazy-loading model, you won’t load the model until the first prediction. This has the benefit of not prematurely loading the model, but it may make the first prediction take slghtly longer.

class MyViewController: UIViewController {
  lazy var model = FritzVisionHumanPoseModelFast()
}

Load model in viewDidLoad

By loading the model in viewDidLoad, you’ll ensure that you’re not loading the model before the view controller is loaded. The model will be ready to go for the first prediction.

class MyViewController: UIViewController {
  let model: FritzVisionHumanPoseModelFast!

  override func viewDidAppear(_ animated: Bool) {
    super.viewDidAppear(animated)
    model = FritzVisionHumanPoseModelFast()
  }
}

Alternatively, you can initialize the model property directly. However, if the ViewController is instantiated by a Storyboard and is the Initial View Controller, the properties will be initialized before the appDelegate function is called. This can cause the app to crash if the model is loaded before FritzCore.configure() is called.

Download the model over the air

Add FritzVision to your Podfile

Include Fritz/Vision in your Podfile.

pod 'Fritz/Vision'

Make sure to run a pod install with the latest changes.

pod install

Download Model

import Fritz

var labelModel: FritzVisionLabelModelFast?

FritzVisionLabelModelFast.fetchModel { model, error in
   guard let downloadedModel = model, error == nil else { return }

   labelModel = downloadedModel
}
@import Fritz;

[FritzVisionLabelModelFast fetchModelWithCompletionHandler:^(FritzVisionLabelModelFast * _Nullable model, NSError * _Nullable error) {
    // Use downloaded label model
}];

2. Create FritzVisionImage

FritzVisionImage supports different image formats.

  • Using a CMSampleBuffer

    If you are using a CMSampleBuffer from the built-in camera, first create the FritzVisionImage instance:

    let image = FritzVisionImage(buffer: sampleBuffer)
    
    FritzVisionImage *visionImage = [[FritzVisionImage alloc] initWithBuffer: sampleBuffer];
    // or
    FritzVisionImage *visionImage = [[FritzVisionImage alloc] initWithImage: uiImage];
    

    The image orientation data needs to be properly set for predictions to work. Use FritzImageMetadata to customize orientation for an image. By default, if you specify FritzVisionImageMetadata the orientation will be .right:

    image.metadata = FritzVisionImageMetadata()
    image.metadata?.orientation = .left
    
    // Add metdata
    visionImage.metadata = [FritzVisionImageMetadata new];
    visionImage.metadata.orientation = FritzImageOrientationLeft;
    

    Note

    Data passed in from the camera will generally need the orientation set. When using a CMSampleBuffer to create a FritzVisionImage the orientation will change depending on which camera and device orientation you are using.

    When using the back camera in the portrait Device Orientation, the orientation should be .right (the default if you specify FritzVisionImageMetadata on the image). When using the front facing camera in portrait Device Orientation, the orientation should be .left.

    You can initialize the FritzImageOrientation with the AVCaptureConnection to infer orientation (if the Device Orientation is portrait):

    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
        let image = FritzVisionImage(sampleBuffer: sampleBuffer, connection: connection)
        ...
    }
    
  • Using an UIImage

    If you are using an UIImage, create the FritzVisionImage instance:

    let image = FritzVisionImage(image: uiImage)
    

    The image orientation data needs to be properly set for predictions to work. Use FritzImageMetadata to customize orientation for an image:

    image.metadata = FritzVisionImageMetadata()
    image.metadata?.orientation = .right
    

    Note

    UIImage can have associated UIImageOrientation data (for example when capturing a photo from the camera). To make sure the model is correctly handling the orientation data, initialize the FritzImageOrientation with the image’s image orientation:

    image.metadata?.orientation = FritzImageOrientation(image.imageOrientation)
    

3. Run image labeling

Run Image Labeling Model

Use the labelModel instance you created earlier to run predictions:

guard let results = try? labelModel.predict(image) else { return }
FritzVisionLabelModelOptions* options = [FritzVisionLabelModelOptions new];
[labelModel predictWithImage:visionImage options:options completion:^(NSArray<FritzVisionLabel* > * _Nullable result, NSError *error) {

  // Code to work with labels here
}];

Configure Label Prediction

Before running image labeling, you can configure the prediction with a FritzVisionLabelModelOptions object.

Settings
imageCropAndScaleOption

.scaleFit (default)

Crop and Scale option for how to resize and crop the image for the model

threshold

0.6 (default)

Confidence threshold for prediction results in the range of [0, 1].

numResults

15 (default)

Maxiumum number of results to return from prediction.

For example, to build a more lenient FritzVisionLabelModelOptions object:

let options = FritzVisionLabelModelOptions()
options.threshold = 0.3
options.numResults = 2

guard let results = try? labelModel.predict(image, options: options) else { return }
FritzVisionLabelModelOptions* options = [FritzVisionLabelModelOptions new];
options.threshold = 0.3;
options.numResults = 2;

[labelModel predictWithImage:image options:options completion:^(NSArray<FritzVisionLabel* > * _Nullable result, NSError *error) {
  // Code to work with labels here
}];

4. Get labels in image

Once you have an array of FritzVisionLabel you can use them to access the image classifications.

 // Created from model prediction.
let labels: [FritzVisionLabel]

// Print highest confidence result
print(labels[0].label)
print(labels[0].confidence)
NSArray<FritzVisionLabel*> * labels;

// Access highest confidence result
results[0].label;
results[0].confidence;