Computer Vision in iOS – Core Camera

Computer Vision on mobile is fun! Here are the few reasons why I personally love computer vision on mobile when compared to traditional desktop based systems.

You need not have to buy a web camera or high resolution camera which should be connected to computer through USB cable.

You generally connect your webcam through a USB cable, so the application you are designing is restricted for testing only inside the circumference of the circle whose radius == length of the cable 😛 .

If you want your system portable you might have to buy a Raspberry Pi or Arduino and connect your webcam to it for doing some processing on the frames it fetches. (My roommates & besties during my bachelors has done some extensive coding on microprocessors and micro controllers, and I clearly know how hard it is.)

If I want to escape the above mentioned step and still want to make my system portable, I literally have to carry the CPU with me 😛

While discussing about the disadvantages of doing CV algorithms on traditional desktop systems you might be already inferring the advantages of mobile based pipelines. Mobiles are easily portable, it is fully equipped with CPU, GPU and various DSP modules which can be utilised based upon the application, and it has a high resolution camera 😉 The only disadvantage with the current mobile computer vision is that you can’t directly take the algorithm you designed that works almost real-time on a computer on to a mobile and expect the same results. Optimisation plays a key role in mobile computer vision. Mobile battery is limited, hence energy usage of your algorithm matters! If you are designing a heavy CV based system, you can’t schedule the whole operations on CPU. You might need to come up with some new strategies that can reduce the CPU usage!

By halting the discussion I started for no specific reason 🙄 , let us get into the topic that this blog is actually dedicated to 😀 .

In this blog, I will be designing an application using Swift and initialise the camera without using the OpenCV. The main idea has been taken inspiration from the following article by Boris Ohayon. In this blog I am developing over his idea and customising it for the applications that I will be designing in future. At any point of this blog, if you are clueless about the Camera pipeline, you can read the article (link provided above) and follow along this tutorial.

Without wasting any more time create a new ‘Single View Application’ with your desired product name and set the language as ‘Swift’.

Add an Image View in the Main.storyboard and reference it in ViewController.swift.

After this, you can build and run the app on your mobile and see that it works like a charm.

So what new thing(s) did I implement in the above code? I converted the code into Swift 3.0 supporting format, added a block through which you can set your FPS from 30 to as high as 240. And did rigorous tests to make sure that the camera pipeline will never go beyond the 10% CPU Usage on the iPhone for any realistic application.

If your application needs higher FPS, you can set it by changing the variable ‘maxFPSDesired. But change it only if you need FPS greater than 30. By default, the FPS will be between 24-30 (fluctuating) and if you want to force the FPS to a fixed number, it won’t be exactly equal to the number you fix and also the CPU usage increases drastically. But if the application you want to try doesn’t have any other costly computations, you can play with higher FPS.

How to count FPS of your app? You can go fancy and code the FPS counter to use in your app. I would suggest you to run the app in profiling mode and choose ‘Core Animation’ in Instruments to check the FPS of your app 😉

//was var finalFormat = AVCaptureDeviceFormat()
var finalFormat = captureDevice.activeFormat //I think this probably isn’t quite right. But are we maybe just initializing it to something with the intent of overriding it in the loop below?
var maxFps: Double = 0
let maxFpsDesired: Double = 30 // was zero, but was that intended?

It builds and runs, but I don’t get any Camera frames, so something must be a bit wrong.

The problem was actually that the delegate signature was no longer correct for Swift 4. This was what I needed:
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
After I made that change, things worked!

I am not able to get it running could you be a bit more specific as to what you did to get it to function.
Did you preserve the var finalFormat = captureDevice.activeFormat and changed only the signature or what?