Blog

ProjectBlog: Recreate Dominos Points for Pies app on iOS with Fritz Image Labeling

Blog: Recreate Dominos Points for Pies app on iOS with Fritz Image Labeling


Dominos is running a campaign to get you to…well, think about Dominos some more. Take a picture of a pizza—any pizza—and get free points towards a free Dominos pie.

How can they do this? Does a person look at every photo taken and determine if it’s pizza or not? Fortunately, they do not:

Domino’s internal teams developed a pizza identifier that will scan each pizza or slice and will then use artificial intelligence-driven software to identify the image as pizza, so that points can be awarded. — press release

In this tutorial, we’ll pretend like we’re part of Dominos’ internal teams and build our own pizza identifier using an on-device image labeling algorithm.

After you’ve completed this tutorial, you’ll be able to:

  • Build a basic camera app in iOS using Swift
  • Use the Fritz iOS SDK to identify the contents of an image
  • Create basic iOS animations using UIKit

Starting out

Download and open the starter project for this tutorial. Choose the starter project.

It’s easiest to setup Fritz using Cocoapods. In the starter project folder, run:

pod repo upadte --sources=master
pod install

This will create a FritzPizzaDetectorDemo.xcworkspace folder. Open this in Xcode.

Build and run the app. Note that since it uses the camera, it won’t work in the Simulator. The app doesn’t do a whole lot to start out—just a plain camera. We’ll be changing that soon.

Fritz Overview

In this tutorial, we’ll be using the Fritz iOS Image Labeling feature to detect pizza. The Fritz SDK comes with a variety of pre-built features that run directly on your phone.

All Fritz Vision APIs use a few constructs:

  • FritzVisionImage: The image that the model runs on. It will wrap the provided pixel buffer or other image you provide.
  • Options: Configuration options that are passed to the model letting you tweak how the model runs.
  • Model: The actual model that runs predictions on the input images.
  • Results: The output of the model. Each predictor has a different type of results. In this tutorial, the results are a list of FritzVisionLabel objects. For style transfer, the results are all CVPixelBuffers.

The Image Labeling feature can label over 1,000 different objects. Fortunately, one of those 1,000 labels is “pizza”.

Setup Fritz Account

Setting up a Fritz Account is easy. Follow the Getting Started directions to setup your Fritz Account and connect the pizza demo to your account. Here are the steps you’ll run through:

  1. Create an account.
  2. Create an iOS App. Make sure that the Bundle ID of your project matches the one you created.
  3. Drag the Fritz-Info.plist file to your project.
  4. Add the Fritz configure to your AppDelegate.

After you run through the initialization steps, build and run your app. When your app has successfully checked in with Fritz, you’ll see the following in the webapp:

Add the Pizza Detector

Open ViewController.swift and add the following at the top of the class:

lazy var labelModel = FritzVisionLabelModel()

This will add initialize the Label Predictor that we’ll use to recognize pizza. Make sure to declare the variable as lazy so that it doesn’t load until the first time you use the model.

Now scroll to the AVCaptureVideoDataOutputSampleBufferDelegate extension. Add the following code to the captureOutput delegate method:

If you run this code, you should see some output in the console. With this code you have:

  1. Created a FritzVisionImage to pass into the Image Labeling predictor. The metadata takes the orientation of the input sampleBuffer into account when running predictions.
  2. Created a FritzVisionLabelModelOptions object with default options. This is not actually necessary right now, but we’ll customize it later.
  3. Run the label model on the created image. The model can throw an error, so we use a guard statement to guard against that and ensure we have predictions going forward.

“But where’s the pizza?” -you (probably) right now. Also me after ordering pizza 30 minutes ago.

Good question. Let’s only look for pizza labels. Update your code to the following:

Here’s what we just changed:

//2: We updated the options. The model was trained on square images that are smaller than images from our camera. Setting the imageCropAndScaleOption to .centerCrop tells the model to take the square of the center of the image. It helps increase prediction accuracy. Also, we lowered the threshold needed to return results.

//3: Nothing new here!

//4: Here we’re filtering the image results for the label “pizza”. The results are of the type [FritzVisionLabel]. Each label has a label and confidence value. We want to filter out all labels that aren’t pizza.

Now we have an app that labels pizza. Let’s add some toppings to our app.

Adding the pizza animation

Now we get to add pizza animations for when we see pizza. Scroll down to the createNewPizzaSlice function at the bottom of the project. Add the following code:

This code creates a UIImageView with the pizza image. It specifies the frame of the view.

Next add the following:

Here’s what we added:

// 2: This adds the new pizza view to the main view of our ViewController. We make sure to bring it to the front of our view. All of our new pizzas will be added on the top.

// 3: Here’s the actual animation. What are we actually doing here? UIView.animate will perform an animation that will last for 2 seconds.

generateRandomPizzaDestination is a function I wrote to pick a point randomly along the edge of the screen. The animation moves the center of the pizza to a random point along the edge.

Once the animation finishes, it removes the pizza view, removing the pizza from our screens (but not our hearts) forever.

Now scroll back up to the captureOutput camera delegate method and add more pizzas as we find them:

Build the app and point your phone at some pizza!

For a bit more fun, add a few more slices on each detection:

Where to go from here?

Download the final project from this tutorial here.

Now you can officially be an engineer at Dominos (disclaimer: not true)! But this tutorial gives you a starting point for building an iOS app using a pre-built Image Labeling model.

Next week, we’ll take it a step further and show you how to build your own custom model to classify different pizza toppings with Google Colab and Fritz Training Templates.


Editor’s Note: Ready to dive into some code? Check out Fritz on GitHub. You’ll find open source, mobile-friendly implementations of the popular machine and deep learning models along with training scripts, project templates, and tools for building your own ML-powered iOS and Android apps.

Join us on Slack for help with technical problems, to share what you’re working on, or just chat with us about mobile development and machine learning. And follow us on Twitter and LinkedIn for all the latest content, news, and more from the mobile machine learning world.

Source: Artificial Intelligence on Medium

Leave a Reply

Your email address will not be published. Required fields are marked *

Back To Top
a

Display your work in a bold & confident manner. Sometimes it’s easy for your creativity to stand out from the crowd.

Social