OpenCV directly in the browser (webassembly + webworker)

We'll see how to use the OpenCV library directly on the browser! To do this, we will compile OpenCV to webassembly and then run it inside a webworker.

What is OpenCV

OpenCV is the most popular library of Computer Vision, and has existed since 1999! What it does is providing a user-friendly and highly efficient development environment. It is a library written in C and C++ by Intel.

OpenCV can also use Intel's embedded performance primitives, a set of low-level routines specific of Intel.

With OpenCV you can develop things like:

2D and 3D feature toolkits

Egomotion estimation

Facial recognition system

Gesture recognition

Human–computer interaction (HCI)

Mobile robotics

Motion understanding

Object identification

Segmentation and recognition

Stereopsis stereo vision: depth perception from 2 cameras

Structure from motion (SFM)

Motion tracking

Augmented reality

Why in the browser

Being able to run computer vision algorithms directly from the browser allows us to move costs to the client device, and thus save many costs on the server.

Imagine you want to get the characteristics of a label of wine from a picture. There are many ways to do this. If we look for the most ergonomic way for our server, we'd move part of the wine label detection logic in the browser. Then, when we fetch the request to the server, we'll only need to send the final vector. This way, we avoid processing the image on the server.

Or even if it's an embedded app for the private use of a company, we could put all the logic in the browser.

Starting a new Next.js project

We're going to use the Next.js framework with React, to ease the setup and use of the project. However, the same can be applied to a project with Angular, Vue.js, Svelte... or vanilla.js.

To start with, let's just create a new Next.js project with the following command:

yarn create next-app

Once you fill in the name of your project, raise the local environment with yarn dev. Now we are ready to start using OpenCV in our Next.js project.

Compile OpenCV into Webassembly

To compile OpenCV to webassembly we can follow the official documentation at:

Using OpenCV in the browser

Now that we have managed to load the OpenCV library in our browser we will see how to run some utilities from the library.

Of course you can do many things with OpenCV. Here I'll show a simple example. Then it will be your job to read the official documentation and learn how to use OpenCV.

The example we're going to use is a simple image processing, to take pictures with the camera and processing them to a grayscale. Although it may seem simple, this is our first "hello world" with OpenCV.

classCV{
// ...previous service code here.../**
* We are going to use the _dispatch event we created before to
* call the postMessage with the msg and the image as payload.
*
* Thanks to what we've implemented in the _dispatch, this will
* return a promise with the processed image.
*/
imageProcessing(payload) {
returnthis._dispatch({ msg: 'imageProcessing', payload })
}
}

Although we have processed the image in a very simple way and we could have done it without using OpenCV, this is our "hello world" with OpenCV. It opens the doors to more complex things.

Conclusion

We have seen how to use the most used library for computer vision in the browser. We've seen how to compile OpenCV into webassembly and use it in a worker to not block the UI for a good performance. I hope that even if you have never heard of this library, now you'll give it a try.

Code

I've uploaded the code of this article on GitHub in case you want to take a look.