Archives

There are many great articles that can get you started on Machine Learning but so many of them focus on a handful examples that have little relevance to the real world. After all, there are only so many cases where one would need to determine the kind of iris based on it’s petals…

As such, one of the biggest challenges is not necessarily understanding Machine Learning – there are plenty good resources out there, but rather finding where to apply the knowledge.

While reading about KNN filtering it occurred to me that it would be a great candidate for repairing images with broken or missing pixels.

But first, what is KNN (or K Nearest Neighbors)? In short, KNN assumes that items with similar characteristics tend to bunch up, and tries to predict the value of an item based on the values of its closest neighbors. I will not go into too many details, because there are many great explanations on the net (one such article can be found here https://www.analyticsvidhya.com/blog/2014/10/introduction-k-neighbours-algorithm-clustering/ ) I will focus, rather, on explaining the rationale behind image repairing.

So why did I think of images? Well, if we look at an image, we see that a lot of the pixels have similar colors to the pixels that are next to them. This is actually an observation that was used when creating some older compression algorithms, anyone remember PCX files? So this means that even if we have a missing pixel, we can make a pretty good guess of its original color just by looking at the pixels that are around it.

Once we have the image in the format that we need, we’ll map it to a dataframe, which, in our case represents a matrix that defines the image. Each pixel is defined by its X and Y coordinates and the color.

Because the Pillow image data defines each pixel as a tuple of 4 values for red, green, blue and alpha each, we will convert it to a single string representation of the format #RRGGBB. In this case I am ignoring the alpha value as I am not working with transparent images, but updating the code to include alpha would be trivial. As a note, the colors are being stored as strings, we would probably have a small performance boost if we stored them as 32bit integers instead. For the sake of simplicity, though, I chose not to do it.

If we’re here, let’s also write the functions to save the dataframe as image files. The save function receives one or more dataframes as parameter then combines the dataframes and writes them to a png file. It will become clearer why I chose to combine dataframes a little bit later on.

Now, let’s simulate that our image is damaged. We’ll do this by randomly removing 30% of the pixels in the image. The simplest way to do this is to take advantage of the methods numpy already gives us. np.random.uniform(min, max, length) creates a list of values between min and max. If the value is greater than 0.7, we make it True, otherwise False. Then we only keep the pixels from the original dataframe for which this value is False. This will keep roughly 70% of the original pixels.

In order to actually see the result, I am filling in the missing pixels with magenta. This is an optional step, just so that we can see what the “broken” image looks like. Let’s save the image to a file so we see what the “corrupt” image looks like.

How do we choose the value of K?

So now to the big question. We know that KNN looks at the K closest neighbours to determine the value of the current pixel. But how do we pick this value? Well, that’s simple… Let’s take the data we have, and split it into two sets. The first one will be the training set, the second one will be the test set. We’ll use 80% of the data as a training set and 20% as a test set. Then, we’ll try to predict the values for the 20% and compare them against the original values for a range of values for K ( I chose between 1 and 20). We’ll then compare the accuracy and take the value of K that gives us the best accuracy.

We can see from here that we get the best results when K is 3 (actually, 3,5,7 are very close, we could pick anyone of those). However, after K=7 we see a decrease in accuracy. That’s most likely because the algorithm is overfitting (i.e. looking at noise in the image and thinking it’s actually a part of the image).

Note 1: I was using an image with a limited number of colors (256). When using a larger palette, the accuracy can get much lower (in the 30s) but the results will still be looking good.

Note 2: Computing the accuracy of K can take a long time for images with lots of colors. For instance, computing the accuracy for a 1280×915 and 32bpp image took 5 hours on my Macbook Pro.

Alright, now that we have the K value, we can actually create the classifier and train it.

And now that we have the prediction, let’s combine the prediction with the original data and voila! We have a complete image… We lost a little bit of the details, when comparing it to the original, but if you didn’t put them side by side it would be hard to spot.

Image with missing pixels restored

I am sure there are ways to improve this, but the results are more than satisfactory.