Computer Vision in iOS – Swift+OpenCV

Hello all, I realised that it has been quite a while since I posted my last blog – Computer Vision in iOS – Core Camera. In my last blog, I discussed about how we can setup Camera in our app without using OpenCV. Since the app has been designed in Swift 3, it is very easy for many budding iOS developers to understand what is going on in that code. I thought of going a step further and design some basic image processing algorithms from scratch. After designing few algorithms, I realised that it is quite hard for me to explain even simple RGB to grayscale conversion without scaring the readers. So, I thought of taking a few steps back and integrate OpenCV into the swift version of our Computer Vision app in hope that it can help the readers during their speed prototyping of proof-of-concepts. But many people have already discussed about how to integrate OpenCV into Swift based apps. The main purpose of this blog post is to introduce you to the data structure of the image and to explain why we are implementing certain things the way they are.

Before starting this blog, it is advised that you read this blog on setting up Core Camera using Swift.

Start by creating a new Xcode Project, select Single View Application. Name your project and organisation, set language as Swift.

For removing some constraints related to UI/UX and since most of the real-time performance apps in Vision either fix to Portrait or Landscape Left/Right orientation through out its usage, go to General -> Deployment Info and uncheck all unnecessary orientations of the app.

Go to Main.storyboard and add the Image View to your app by drag-and-drop from the following menu to the storyboard.

Go to “Show the Size Inspector” on the top-right corner and make the following changes.

Now add some constraints to the Image View.

After the above settings, you can observe that the Image View fills the whole screen on the app. Now go to ‘Show the attributes inspector’ on the top right corner and change ‘Content Mode’ from Scale To Fill to ‘Aspect Fill’.

Now add an IBOutlet to the ImageView in ViewController.swift file. Also add the new swift file named ‘CameraBuffer.swift’ file and copy paste the code shown in the previous blog. Also change your ViewController.swift file as shown in previous blog. Now if you run your app, you can see a portrait mode camera app with ~30 FPS. (Note: Don’t forget to add permissions to use camera in Info.plist).

Let us dive into adding OpenCV into our app. First let us add the OpenCV Framework into our app. If you are following my blogs from starting, it should be easy for you.

Let us get into some theoretical discussion. (Disclaimer: It is totally fine to skip this bullet point if you only want the app working). What is an Image? From the signals and systems perspective, an Image is defined as a 2D discrete signal where each pixel signifies a value between 0-255 representing a specific gray level (0 represents black and 255 corresponds to white). To understand this better refer to the picture shown below (PC: Link). Now you might be wondering what is adding color to the image if each pixel is storing only the gray values. If you observe any documentation online you can see that the color image is actually referred as RGB image or RGBA image. The R,G, B in RGB image refers to the Red, Green and Blue Channels of the image and where each channel corresponds to the 2D grayscale signal with values between 0-255. The A channel in RGBA image represents the alpha channel or the opacity of that pixel. In OpenCV, the image is generally represented as a Matrix in BGR or BGRA format. In our code, we are getting access to the every single frame captured by camera in UIImage format. Hence, in order to do any image processing on these images we have to convert them from UIImage to cv::Mat and do all the processing that is required and send them back as UIImage to view it on the screen.

Add a new file -> ‘Cocoa Touch Class’, name it ‘OpenCVWrapper’ and set language to Objective-C. Click Next and select Create. When it prompted to create bridging header click on the ‘Create Bridging Header’ button. Now you can observe that there are 3 files created with names: OpenCVWrapper.h, OpenCVWrapper.mm, and -Bridging-Header.h. Open ‘-Bridging-Header.h’ and add the following line: #import “OpenCVWrapper.h”

Go to ‘OpenCVWrapper.h’ file and add the following lines of code. In this tutorial, let us do the simple RGB to Grayscale conversion.