Training models with Fritz AI

Dataset Collection Header

Fritz AI Model Training allows developers to use datasets they’ve generated and collected to train production-ready mobile machine learning models — without any code. It also makes optimization simple and straightforward, with variants available for a variety of project needs and use cases. Both Core ML and TensorFlow Lite models are created automatically, making both iOS and Android development easier.

Because Model Training exists within the same workflow as both the Dataset Generator and the Dataset Collection System, developers never have to leave Fritz AI to build and scale production-quality models.

Core Concepts

Training a machine learning model can be overwhelming, even for the most seasoned ML professionals. To help, we've done the hard work optimizing models for mobile use and boiled things down to just a few easy steps.

Below are some core concepts to help you get started.

Model Type

The Model Type refers to the task the model will perform. For example, an Object Detection model identifies objects in an image and draws a bounding box around them. A Pose Estimation model predicts the location of specific keypoints in an image (e.g. nose and mouth). You'll need to make sure you know which model type is right for your use case ahead of time, as everything from the data you label to the post-processing code you implement depends on it.


Datasets are the inputs to your model. They contain images and annotations your model learns from. You can select one or more datasets to train a model. It's important the your dataset have annotations for the Model Type you are training. For example, if you are training an Object Detection model, annotations in your dataset must include bounding boxes around objects.

Model Variants

Model Variants define specific model configurations designed for different use cases. The "small" variant produces tiny models that make your application bundle smaller and don't use a lot of user data. The "fast" variant is designed for speed, when you need to run your model in real-time or on video. The "accurate" model is optimized for prediction quality. While we strive to train models that are both small, fast, and accurate, there are always tradeoffs. Model Variants let you prioritize the performance the matters most to you.


Checkpoints are snapshots of a model at a given time. A checkpoint in each mobile format is automatically saved when your model is finished training so you can integrate it into your app. When configuring training jobs, you have the option of saving these checkpoints as new versions of existing models stored in Fritz AI. You can also use existing checkpoints to pick up where a previous training job left off. When you start from an existing model the new one will usually train faster and retain some (but not all) knowledge from the previous model. Starting from scratch is recommended if you radically change the type of data you’re showing the model.

Training a model

From the training tab, click the TRAIN NEW MODEL button.

Train New Job

1. Configure your training job

Create a training job

First, provide a training job name you can reference later. Next, choose a model type. After you've chosen a model type, select one or more datasets to train your model with. Datasets must have annotations matching the model type you have selected. For example, Object Detection models must be trained with data that have bounding box annotations.


If you select more than one dataset for training, make sure their annotation configurations match exactly. For example, if you are training a model to detect cars, both datasets must have a single object named "car".

With datasets selected, choose the model variant you wish to train. Each variant optimizes for different needs: small, fast, or accurate models.

You can then choose a filename for your resulting models. Note that model filenames are used as class names in your mobile app so make sure that they conform to code syntax and style guidelines (i.e. camel case).

Next, provide a training time budget. In general, the longer you train a model, the more accurate it will be. If you don't know how long to train your model, you can use the number recommended. If you model stops improving before this budget has been exceeded, we will automatically stop training and you will not be charged for the remaining hours. You can always continue training from an existing checkpoint later.

Optional checkpoint configuration

Optional: Choose an existing model from which to start a new training job. You can also save output checkpoints as versions of models you've already built.

Finally, there are two optional configuration parameters. If you would like to start training from an existing model checkpoint, select the name and version from the drop down menus. Note that the model you select must be of the same type and variant and take in the same dataset. For example, if you are training a fast variant of an object detection model, the existing checkpoint you select must also be a fast object detection model, and it must predict the same number of objects.

While existing checkpoints provide a starting point for your model, output checkpoints control where newly-trained models are stored. By default, checkpoints will be saved to a brand new model in your Fritz AI account. However, you can choose to save new models as versions of an existing model if you are already working on a model in development. Saving checkpoints as new versions of an existing model makes it possible to deploy your newly-trained model without needing to re-compile and install any application code.

When you've configured your model training job, click the NEXT button and confirm the details.

2. Monitoring your training job

Training job details

The training job details page provides information on the progress of your training job. The status indicates if the job is in progress or has stopped for any reason. You can also view the specific configuration used for each training job to keep track of all of your experiments.

The final section of the training job details page is a list of Output Checkpoints. These checkpoints are the output of your model training job for each platform. Select a row to view model details. When training has completed, new checkpoints will automatically be uploaded as versions of these models.

Output model checkpoints from your training job.
Output model checkpoints from your training job.

When you start a new training job, untrained model checkpoints are created. These checkpoints are untrained models and will not produce accurate predictions. However, they are useful for setting up your mobile app while your model is training.

Finally, you can also cancel running jobs at any time.

3. Deploying your model

Models training within the Fritz webapp are designed to work seamlessly with the Fritz SDK. To deploy your model within a mobile app, select a model output checkpoint to open the Model Details page. Click the SDK instructions button to receive platform-specific integration instructions. If you don't yet have an app or would like a simple demo project for fast iteration, we have provided a number of example projects on GitHub.

Add Fritz AI to your own training pipeline with the Python SDK

If you already have your own training pipeline, you can use the Fritz Python SDK.