Which image processing techniques could be used to implement an application that detects the christmas trees displayed in the following images?

I'm searching for solutions that are going to work on all these images. Therefore, approaches that require training haar cascade classifiers or template matching are not very interesting.

I'm looking for something that can be written in any programming language, as long as it uses only Open Source technologies. The solution must be tested with the images that are shared on this question. There are 6 input images and the answer should display the results of processing each of them. Finally, for each output image there must be red lines draw to surround the detected tree.

How would you go about programmatically detecting the trees in these images?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

3

Are we allowed to use some of the images for training, or should all the supplied images be used for validation? Either way, cool competition :D
– Hannes OvrénDec 25 '13 at 13:31

7

@karlphillip, do you want us to use these images for testing and other images for training? It's just that it's not clear what the training set is.
– GilLeviDec 25 '13 at 16:47

15

@karlphillip: My advice: drop the "open source" requirement. It really doesnt matter what language/framework you use. Image-processing/computer-vision algorithms are language agnostic, so if you can write it in MATLAB, you can certainly do it OpenCV or any other framework you prefer... Also I'm still not clear what you consider training/testing images!
– AmroDec 29 '13 at 11:02

2

@karlphillip thanx for mobilizing all of us to contribute to this 'quest' of yours! It has been a great opportunity to spend some hours productively, but most importantly, to get to see how many different approaches can be found to a single problem... Hope you do it again for the 1st of Jan (maybe a 'sleigh of Santa Claus' challenge? ;-))
– sepdekDec 30 '13 at 14:04

2

OK, I reworded the question to remove the competition elements. I think that should allow it to stand on its own just fine.
– Brad Larson♦Jan 4 '14 at 20:30

10 Answers
10

I have an approach which I think is interesting and a bit different from the rest. The main difference in my approach, compared to some of the others, is in how the image segmentation step is performed--I used the DBSCAN clustering algorithm from Python's scikit-learn; it's optimized for finding somewhat amorphous shapes that may not necessarily have a single clear centroid.

At the top level, my approach is fairly simple and can be broken down into about 3 steps. First I apply a threshold (or actually, the logical "or" of two separate and distinct thresholds). As with many of the other answers, I assumed that the Christmas tree would be one of the brighter objects in the scene, so the first threshold is just a simple monochrome brightness test; any pixels with values above 220 on a 0-255 scale (where black is 0 and white is 255) are saved to a binary black-and-white image. The second threshold tries to look for red and yellow lights, which are particularly prominent in the trees in the upper left and lower right of the six images, and stand out well against the blue-green background which is prevalent in most of the photos. I convert the rgb image to hsv space, and require that the hue is either less than 0.2 on a 0.0-1.0 scale (corresponding roughly to the border between yellow and green) or greater than 0.95 (corresponding to the border between purple and red) and additionally I require bright, saturated colors: saturation and value must both be above 0.7. The results of the two threshold procedures are logically "or"-ed together, and the resulting matrix of black-and-white binary images is shown below:

You can clearly see that each image has one large cluster of pixels roughly corresponding to the location of each tree, plus a few of the images also have some other small clusters corresponding either to lights in the windows of some of the buildings, or to a background scene on the horizon. The next step is to get the computer to recognize that these are separate clusters, and label each pixel correctly with a cluster membership ID number.

For this task I chose DBSCAN. There is a pretty good visual comparison of how DBSCAN typically behaves, relative to other clustering algorithms, available here. As I said earlier, it does well with amorphous shapes. The output of DBSCAN, with each cluster plotted in a different color, is shown here:

There are a few things to be aware of when looking at this result. First is that DBSCAN requires the user to set a "proximity" parameter in order to regulate its behavior, which effectively controls how separated a pair of points must be in order for the algorithm to declare a new separate cluster rather than agglomerating a test point onto an already pre-existing cluster. I set this value to be 0.04 times the size along the diagonal of each image. Since the images vary in size from roughly VGA up to about HD 1080, this type of scale-relative definition is critical.

Another point worth noting is that the DBSCAN algorithm as it is implemented in scikit-learn has memory limits which are fairly challenging for some of the larger images in this sample. Therefore, for a few of the larger images, I actually had to "decimate" (i.e., retain only every 3rd or 4th pixel and drop the others) each cluster in order to stay within this limit. As a result of this culling process, the remaining individual sparse pixels are difficult to see on some of the larger images. Therefore, for display purposes only, the color-coded pixels in the above images have been effectively "dilated" just slightly so that they stand out better. It's purely a cosmetic operation for the sake of the narrative; although there are comments mentioning this dilation in my code, rest assured that it has nothing to do with any calculations that actually matter.

Once the clusters are identified and labeled, the third and final step is easy: I simply take the largest cluster in each image (in this case, I chose to measure "size" in terms of the total number of member pixels, although one could have just as easily instead used some type of metric that gauges physical extent) and compute the convex hull for that cluster. The convex hull then becomes the tree border. The six convex hulls computed via this method are shown below in red:

The source code is written for Python 2.7.6 and it depends on numpy, scipy, matplotlib and scikit-learn. I've divided it into two parts. The first part is responsible for the actual image processing:

@stachyra I also thought about this approach before proposing my simpler ones. I think this has a great potential to be extended and generalized to produce good results in other cases also. You could experiment with neural nets for clustering. Something like a SOM or neural gas would do excellent work. Nevertheless, great proposal and thumbs up from me!
– sepdekDec 31 '13 at 9:47

4

@Faust & Ryan Carlson: thanks, guys! Yes, I agree that the upvote system, while it works well for adjudicating between 2 or 3 short answers all submitted within a few hours of each other, has serious biases when it comes to contests with long answers that play out over extended periods of time. For one thing, early submissions begin accumulating upvotes before later ones are even available for public review. And if answers are all lengthy, then as soon as one establishes a modest lead, there is often a "bandwagon effect" as people only upvote the first one without bothering to read the rest.
– stachyraJan 1 '14 at 19:37

2

@stachyra great news friend! Warmest congrats and may this mark a beginning for your new year!
– sepdekJan 4 '14 at 9:41

1

@lennon310: I haven't tried a local maximum detection filter yet on this problem, but if you want to explore it yourself, scipy includes this one. My Python source code for this project was so short that I was actually able to publish 100% of it; literally all you would need to do is copy-and-paste my two code snippets into separate .py files and then substitute a call to scipy.ndimage.filters.maximum_filter() in the same place where I had used a threshold.
– stachyraJan 4 '14 at 19:22

EDIT NOTE: I edited this post to (i) process each tree image individually, as requested in the requirements, (ii) to consider both object brightness and shape in order to improve the quality of the result.

Below is presented an approach that takes in consideration the object brightness and shape. In other words, it seeks for objects with triangle-like shape and with significant brightness. It was implemented in Java, using Marvin image processing framework.

The first step is the color thresholding. The objective here is to focus the analysis on objects with significant brightness.

In the second step, the brightest points in the image are dilated in order to form shapes. The result of this process is the probable shape of the objects with significant brightness. Applying flood fill segmentation, disconnected shapes are detected.

As shown in the output image, multiple shapes was detected. In this problem, there a just a few bright points in the images. However, this approach was implemented to deal with more complex scenarios.

In the next step each shape is analyzed. A simple algorithm detects shapes with a pattern similar to a triangle. The algorithm analyze the object shape line by line. If the center of the mass of each shape line is almost the same (given a threshold) and mass increase as y increase, the object has a triangle-like shape. The mass of the shape line is the number of pixels in that line that belongs to the shape. Imagine you slice the object horizontally and analyze each horizontal segment. If they are centralized to each other and the length increase from the first segment to last one in a linear pattern, you probably has an object that resembles a triangle.

The advantage of this approach is the fact it will probably work with images containing other luminous objects since it analyzes the object shape.

Merry Christmas!

EDIT NOTE 2

There is a discussion about the similarity of the output images of this solution and some other ones. In fact, they are very similar. But this approach does not just segment objects. It also analyzes the object shapes in some sense. It can handle multiple luminous objects in the same scene. In fact, the Christmas tree does not need to be the brightest one. I'm just abording it to enrich the discussion. There is a bias in the samples that just looking for the brightest object, you will find the trees. But, does we really want to stop the discussion at this point? At this point, how far the computer is really recognizing an object that resembles a Christmas tree? Let's try to close this gap.

That's interesting. I hope you can get the same results when each image is processed individually. I edited the question 4hrs previously to you posting the answer to state this specifically. It would be awesome if you could update your answer with these results.
– karlphillipDec 26 '13 at 15:20

@Marvin in your triangle detection, how did you handle the fluctuation of mass? It is not a strict triangle, the mass is not mono as y changes
– user3054997Dec 31 '13 at 4:43

2

@user3054997: That's another point. As I posted, the algorithm does not seeks for the strict triangle shapes. It analyze each object and consider a tree that ones that "resembles" a triangle with a simple criteria: the mass of the object is used to increases as y increase and the center of the mass of each horizontal object segment is almost centralized to each other.
– Gabriel Ambrósio ArchanjoDec 31 '13 at 12:11

@Marvin My solution is really simple, I stated it in my answer too. By the way it worked better than your first solution. If I recall correctly, in your first answer, you talked about feature descriptors to detect small light texture, which is not what you are doing here. I simply said that your current approach and results are much more similar to mine than to your first solution. Of course I do not expect you to admit it, I stated it just for the record.
– smesoDec 31 '13 at 13:04

1

@sepdek There is here a couple of solutions that are genuinely much better than mine and they are still getting half of my upvotes. There is nothing wrong in "getting inspired" by other solutions. I saw your solutions too, I have nothing to say against you, you posted them after me and my "idea" was not so original to say that you just copied me. But Marvin was the only one who posted before me and edited is solution after seeing mine using the same algorithm... at least he could have said "Yeah, I liked your solution and I reused it" there is nothing wrong, it is just a game.
– smesoJan 3 '14 at 12:57

The first step is to detect the most bright pixels in the picture, but we have to do a distinction between the tree itself and the snow which reflect its light. Here we try to exclude the snow appling a really simple filter on the color codes:

Now we have almost done, but there are still some imperfection due to the snow.
To cut them off we'll build a mask using a circle and a rectangle to approximate the shape of a tree to delete unwanted pieces:

Hello! Make sure your answer follows all the requirements: There are 6 input images and the answer should display the results of processing each of them;.
– karlphillipDec 26 '13 at 17:07

Hi! You can pass filenames as CLI arguments to my program: ./christmas_tree ./*.png. They can be as many as you want, the results will be showed one after the other pressing any key. Is this wrong?
– smesoDec 26 '13 at 17:13

It's OK, but you still need to upload the images and share them in your question so the viewers of the thread can actually see your result. Letting people see what you did will improve your chances of getting up votes ;)
– karlphillipDec 26 '13 at 17:18

I'm trying to find a solution for this, I have some connectivity problems.
– smesoDec 26 '13 at 17:19

2

Great! Now you can rescale them inside the answer with the following code: <img src="http://i.stack.imgur.com/nmzwj.png" width="210" height="150"> Just change the link to the picture ;)
– karlphillipDec 26 '13 at 17:45

Besides the feature in color space, I also used texture feature that is relevant with the
neighborhood rather than each pixel itself. Here I linearly combined the intensity from the
3 original channels (R,G,B). The reason why I formatted this way is because the christmas
trees in the picture all have red lights on them, and sometimes green/sometimes blue
illumination as well.

I applied a 3X3 local binary pattern on I0, used the center pixel as the threshold, and
obtained the contrast by calculating the difference between the mean pixel intensity value
above the threshold and the mean value below it.

Since I have 4 features in total, I would choose K=5 in my clustering method. The code for
k-means are shown below (it is from Dr. Andrew Ng's machine learning course. I took the
course before, and I wrote the code myself in his programming assignment).

Since the program runs very slow in my computer, I just ran 3 iterations. Normally the stop
criteria is (i) iteration time at least 10, or (ii) no change on the centroids any more. To
my test, increasing the iteration may differentiate the background (sky and tree, sky and
building,...) more accurately, but did not show a drastic changes in christmas tree
extraction. Also note k-means is not immune to the random centroid initialization, so running the program several times to make a comparison is recommended.

After the k-means, the labelled region with the maximum intensity of I0 was chosen. And
boundary tracing was used to extracted the boundaries. To me, the last christmas tree is the most difficult one to extract since the contrast in that picture is not high enough as they are in the first five. Another issue in my method is that I used bwboundaries function in Matlab to trace the boundary, but sometimes the inner boundaries are also included as you can observe in 3rd, 5th, 6th results. The dark side within the christmas trees are not only failed to be clustered with the illuminated side, but they also lead to so many tiny inner boundaries tracing (imfill doesn't improve very much). In all my algorithm still has a lot improvement space.

Some publications indicates that mean-shift may be more robust than k-means, and many
graph-cut based algorithms are also very competitive on complicated boundaries
segmentation. I wrote a mean-shift algorithm myself, it seems to better extract the regions
without enough light. But mean-shift is a little bit over-segmented, and some strategy of
merging is needed. It ran even much slower than k-means in my computer, I am afraid I have
to give it up. I eagerly look forward to see others would submit excellent results here
with those modern algorithms mentioned above.

Yet I always believe the feature selection is the key component in image segmentation. With
a proper feature selection that can maximize the margin between object and background, many
segmentation algorithms will definitely work. Different algorithms may improve the result
from 1 to 10, but the feature selection may improve it from 0 to 1.

I didn't test yet, but I think it's no problem:)
– lennon310Dec 27 '13 at 13:53

@lennon310 I think if you drop the boundaries and get the convex hull you will get rid of the holes problem. Remember that convex hull is the smallest area that includes all points in a set.
– sepdekJan 3 '14 at 8:51

This is my final post using the traditional image processing approaches...

Here I somehow combine my two other proposals, achieving even better results. As a matter of fact I cannot see how these results could be better (especially when you look at the masked images that the method produces).

At the heart of the approach is the combination of three key assumptions:

Images should have high fluctuations in the tree regions

Images should have higher intensity in the tree regions

Background regions should have low intensity and be mostly blue-ish

With these assumptions in mind the method works as follows:

Convert the images to HSV

Filter the V channel with a LoG filter

Apply hard thresholding on LoG filtered image to get 'activity' mask A

Great stuff! Please make sure your other answers also follow this format. To compete for the bounty you must use an open source technology, and unfortunately Matlab is not one of them. However, SciLab and Octave are and they provide similar syntax and functions. ;)
– karlphillipDec 27 '13 at 20:52

@karlphillip Somehow this question ended up having a Matlab tag. If open source is really a must I would recommend removing it.
– Dennis JaheruddinDec 30 '13 at 13:07

@sepdek Very nice, perhaps something could still be done to include the 'holes' in the final picture. (Add all pixels that are completely surrounded by approved pixels?!)
– Dennis JaheruddinDec 30 '13 at 13:09

1

@karlphillip thanx man! I am glad you found my approach interesting. In addition I would like to congratulate you for selecting the most elegant solution and not the one with the most votes!!!
– sepdekJan 5 '14 at 10:40

Detect eges in computed roi. Tree has a lot of edges (middle right image)

Dilate result

Erode with bigger radius ( bottom left image)

Select the biggest (by area) object - it's the result region

ConvexHull ( tree is convex polygon ) ( bottom right image )

Bounding box (bottom right image - grren box )

Step by step:

The first result - most simple but not in open source software - "Adaptive Vision Studio + Adaptive Vision Library":
This is not open source but really fast to prototype:

Whole algorithm to detect christmas tree (11 blocks):

Next step. We want open source solution. Change AVL filters to OpenCV filters:
Here I did little changes e.g. Edge Detection use cvCanny filter, to respect roi i did multiply region image with edges image, to select the biggest element i used findContours + contourArea but idea is the same.

Alright, thanks. I tested it and it is beautiful. As soon as this question is reopened (other users have to help me with that) I can set another bounty to reward you. Congratulations!
– karlphillipJan 4 '14 at 15:22

everything with Hues (H) between 210 - 320 degrees is discarded as blue-magenta that is supposed to be in the background or in non-relevant areas

everything with Values (V) lower that 40% is also discarded as being too dark to be relevant

Of course one may experiment with numerous other possibilities to fine-tune this approach...

Here is the MATLAB code to do the trick (warning: the code is far from being optimized!!! I used techniques not recommended for MATLAB programming just to be able to track anything in the process-this can be greatly optimized):

Results:

Hello, thanks for the answer. Please take a moment to read the Requirements section to make sure your answer follows all the instructions. You forgot to share the resulting images. ;)
– karlphillipDec 27 '13 at 13:06

2

@karlphillip sepdek don't have enough reputation to share images, I moved images into answer body according to his link and instructions. Not sure though, that those are correct ones, feel free to comment this part.
– alkoDec 27 '13 at 13:29

@alko I know, thanks. But some of the images you shared were not in the input set. The answer must show the result of processing all 6 images shared on the question.
– karlphillipDec 27 '13 at 13:30

@karlphillip that's his images, not mine. that is exatly what I meant by "comment this part" ;)
– alkoDec 27 '13 at 13:31

2

Sorry for causing issues...not my intention. I have included all images in the initial dataset and enhanced it with even more just to prove that my concept is robust...
– sepdekDec 27 '13 at 13:47

Some old-fashioned image processing approach...
The idea is based on the assumption that images depict lighted trees on typically darker and smoother backgrounds (or foregrounds in some cases). The lighted tree area is more "energetic" and has higher intensity.
The process is as follows:

I am a noob here so I cannot upload images. Please see results on the provided links in my description.
– sepdekDec 27 '13 at 12:21

Ok, but you still have to use the images shared on the question like everybody else is doing. Once you process them, upload it somewhere and edit your answer to add the links. Later I'll edit your answer and place the images inside it for you.
– karlphillipDec 27 '13 at 13:08

Using a quite different approach from what I've seen, I created a php script that detects christmas trees by their lights. The result ist always a symmetrical triangle, and if necessary numeric values like the angle ("fatness") of the tree.

The biggest threat to this algorithm obviously are lights next to (in great numbers) or in front of the tree (the greater problem until further optimization).
Edit (added): What it can't do: Find out if there's a christmas tree or not, find multiple christmas trees in one image, correctly detect a cristmas tree in the middle of Las Vegas, detect christmas trees that are heavily bent, upside-down or chopped down... ;)

The different stages are:

Calculate the added brightness (R+G+B) for each pixel

Add up this value of all 8 neighbouring pixels on top of each pixel

Rank all pixels by this value (brightest first) - I know, not really subtle...

Choose N of these, starting from the top, skipping ones that are too close

Calculate the median of these top N (gives us the approximate center of the tree)

Start from the median position upwards in a widening search beam for the topmost light from the selected brightest ones (people tend to put at least one light at the very top)

From there, imagine lines going 60 degrees left and right downwards (christmas trees shouldn't be that fat)

Decrease those 60 degrees until 20% of the brightest lights are outside this triangle

Find the light at the very bottom of the triangle, giving you the lower horizontal border of the tree

Done

Explanation of the markings:

Big red cross in the center of the tree: Median of the top N brightest lights