Recognizing license plates from video

The following is written in c with the use of the very handy opencv library.

So I wanted to identify the license plate numbers of cars that drive by my home. Set up a camera with a telephoto lens connected to a simple self made motion detection program that would capture images when there is enough movement in the video. Let it capture for an hour or so and see what it received (hopefully no privacy issues) :

License plate 1

License plate 2

Now I want to automatically obtain the license plates from these images. First convert to greyscale:

Now the first problem is detecting the license plate location. I want to find the white rectangle. The problem is that the image contains a lot of stuff that’s much whiter. But the plate itself has the advantage that it’s relatively whiter than its surroundings. To help a bit, I’m using a technique also common for HDR photography, normalize the image to the local brightness. This can be done by substracting a smoothed version from the image. I smoothed with a rectangular 251×101 gaussian convolution filter as the plate is also rectangular and mostly horizontal in the image. The substracted image is as follows:

Source image minus smoothed version

This image can now be thresholded:

Thresholded

And this can be used to run a contour detection over. Opencv also has a handy polygon approximation function that can be run on the contour data. Using this we get a lot of polygons for the above image. Filtering out the polygons that only have 4 vertices and then calculating the angle of each vertice to determine if they are sufficiently close to 90 degrees we can filter out most of the unwanted stuff. In this case, only the numberplate remains:

Now the points have to be ordered. Finding the vertex with the lowest x+y value will give us the left-top corner. Then determining the rotation of the rectangle (clockwise or counter-clockwise) by using the crossproduct of two edges. When the points are re-ordered the licenseplate can be scanned from left to right and top to bottom to copy it onto a nicely rotated version:

Rotated and scaled license plate

Unfortunately it becomes clear that the resolution might be too low for accurate OCR (text recognition). Let’s try anyway. Substracting a smoothed version again and thresholding gives us this:

Thresholded license plate 1

But running it through freeocr outputs “(1 Hi 24\\1”. Unfortunately, I don’t know how to get any better results without trying to completely write an OCR program myself.

Running the second example through the whole automated process gives us this thresholded image:

Thresholded license plate 2

Which is recognized as “lYEU@” for some reason. I’ve also tried the online ocr service at http://www.onlineocr.net/default.aspx. Which resulted in “rzRI 24W” and “EYE174384′”. Oddly a better result for the second image and a worse on for the first.

So, the biggest improvement would be either to get a better video camera, or a better OCR solution. Maybe one that’s more geared towards number plates. Or one where you can enter your own allowed string patterns.