MIT algorithm aims to eradicate reflections from photos taken through windows

MIT researchers have created a new algorithm that, in a broad range of cases, can automatically remove reflections from digital photos. On the left is the original photo taken through a window, with the photographer's reflection clearly visible. On the right, the reflection has been separated from the photo. Courtesy of the researchers

Researchers at the Massachusetts Institute of Technology claim to have developed a method for eliminating glass reflections in photos via digital processing. It is hoped that with further development the idea could see its way into cameras themselves, enabling reflections to be automatically removed when they interfere with the view through a window.

The new algorithm, due to be formally announced in June at the Boston Computer Vision and Pattern Recognition conference, is claimed to offer a way of distinguishing the image reflected by a window surface from the image of whatever is behind the window. Once the two images are clearly identified one can be subtracted from the whole to leave the other.

MIT researchers noted that objects reflected by glass surfaces tend to have two reflections, not one, and that the two images of reflected objects are slightly off-set. The multiple images in a reflection come about sometimes because windows are double glazed for insulation and thus have two panes of glass, and other times because light reflects from the rear surface of a pane of glass as well as the facing surface.

Further research showed that the interference of a reflection can often be identified by the way it interrupts the regular shape and color patterns of natural and man-made objects on the other side of the glass. Working with small clusters of 8x8 pixels, researchers analyzed color and tonal relationships within the group and came to a method by which certain data correlations indicated that the group included detail from a reflection.

The result is that when an image of, for example, a view shot through a bus window is analyzed, the team can separate the image of the view from the reflected image of the photographer that shot the view – and then can show either the view on its own, or the photographer on his or her own.

If successful the algorithm may have serious consequences for the flexible rubber lens hood market, though early indications suggest polarizing filter manufacturers will be safe for some time to come.

Removing reflections from photos taken through windows

New algorithm exploits multiple reflections in individual images to distinguish reflection from transmission.

Larry Hardesty | MIT News OfficeMay 11, 2015

It's hard to take a photo through a window without picking up reflections of the objects behind you. To solve that problem, professional photographers sometimes wrap their camera lenses in dark cloths affixed to windows by tape or suction cups. But that's not a terribly attractive option for a traveler using a point-and-shoot camera to capture the view from a hotel room or a seat in a train.

At the Computer Vision and Pattern Recognition conference in June, MIT researchers will present a new algorithm that, in a broad range of cases, can automatically remove reflections from digital photos. The algorithm exploits the fact that photos taken through windows often feature two nearly identical reflections, slightly offset from each other.

“In Boston, the windows are usually double-paned windows for heat isolation during the winter,” says YiChang Shih, who completed his PhD in computer science at MIT this spring and is first author on the paper. “With that kind of window, there's one reflection coming from the inner pane and another reflection from the outer pane. But thick windows are usually enough to produce a double reflection, too. The inner side will give a reflection, and the outer side will give a reflection as well.”

Without the extra information provided by the duplicate reflection, the problem of reflection removal is virtually insoluble, Shih explains. “You have an image from outdoor and another image from indoor, and what you capture is the sum of these two pictures,” he says. “If A+B is equal to C, then how will you recover A and B from a single C? That's mathematically challenging. We just don't have enough constraints to reach a conclusion.”

Thinning the field

The second reflection imposes the required constraint. Now the problem becomes recovering A, B, and C from a single D. But the value of A for one pixel has to be the same as the value of B for a pixel a fixed distance away in a prescribed direction. That constraint drastically reduces the range of solutions that the algorithm has to consider.

Nonetheless, a host of solutions still remain. To home in on one of them, Shih and his coauthors — professors of computer science and engineering Frédo Durand and Bill Freeman, who were Shih's thesis advisors, and Dilip Krishnan, a former postdoc in Freeman's group who's now at Google Research — assume that both the reflected image and the image captured through the window have the statistical regularities of so-called natural images.

The basic intuition is that at the level of small clusters of pixels, in natural images — unaltered representations of natural and built environments — abrupt changes of color are rare. And when they do occur, they occur along clear boundaries. So if a small block of pixels happens to contain part of the edge between a blue object and a red object, everything on one side of the edge will be bluish, and everything on the other side will be reddish.

In computer vision, the standard way to try to capture this intuition is with the notion of image gradients, which characterize each block of pixels according to the chief direction of color change and the rate of change. But Shih and his colleagues found that this approach didn't work very well.

Playing the odds

Ultimately, they settled on a new technique co-developed by Daniel Zoran, a postdoc in Freeman's group. Zoran and Yair Weiss of the Hebrew University of Jerusalem created an algorithm that divides images into 8-by-8 blocks of pixels; for each block, it calculates the correlation between each pixel and each of the others. The aggregate statistics for all the 8-by-8 blocks in 50,000 training images proved a reliable way to distinguish reflections from images shot through glass.

In their paper, Shih and his colleagues report performing searches on Google and the Flickr photo database using terms like “window reflection photography problems.” After excluding results that weren't photos shot through glass, they had 197 images, 96 of which exhibited double reflections that were offset far enough for their algorithm to work.

“People have worked on methods for eliminating these reflections from photos, but there had been drawbacks in past approaches,” says Yoav Schechner, a professor of electrical engineering at Israel's Technion. “Some methods attempt using a single shot. This is very hard, so prior results had partial success, and there was no automated way of telling if the recovered scene is the one reflected by the window or the one behind the window. This work does a good job on several fronts.”

“The ideas here can progress into routine photography, if the algorithm is further robustified and becomes part of toolboxes used in digital photography,” he adds. “It may help robot vision in the presence of confusing glass reflection.”

That was Ed Thorp, famous author of "beat the dealer" and his follow-up "beat the market". He's a legend on wall street, as are many other MIT alum such as Jim Simmons.

MIT attracts the brightest and makes them even smarter. Other schools are known for their style, panache, and relationships. MIT is known for its brains. Not only is it the best engineering school in the world, its in a class by itself.

Yes, MIT alum Jim Simmons is another example of "MIT is known for its brains":

"Simons received his doctorate at 23; advanced code breaking for the National Security Agency at 26; led a university math department at 30; won geometry's top prize at 37; founded Renaissance Technologies, one of the world's most successful hedge funds, at 44".

You need "brain power" to beat highly regulated/monitored system without breaking the law such as; Casinos, Lottery, Stock market...

Forensic work is one good application. Reflection reduction (NOT elimation) is another useful on, where you reduce the reflection of say, a bride looking out the windows to maintain the focus of attention while creating the mood. I'm not a big fan of polarizing filters as sometimes they affect the global look of a scene. But most importantly, even with my Hoyas, it affects the edge sharpness of my photos. I know a lot say its a waste of time, you just see the smaller picture at the moment.

I'd suggest some people posting uber-smart comments here to re-read the above quote again and again until you understand it.

Didn't you ever take a quick shot through a window of an excursion bus of something happening outside, only to find out later that the shot is pretty much ruined by reflections? Oh, you probably don't have time for that, too busy posting smart comments on forums... OK, I can understand that. Carry on.

...or you could always learn how to shoot through windows by a) opening it or b) holding your camera right up against the glass. Either of these options will give you MUCH better results than this 'we got nothin' better to do' waste of research.

You obviously have no experience in research. Often students in maths and engineering will work with fundamental or abstract questions and problems that may have no present technologial use.

Students and researches understanding computer algorithms and developing new algorithms is never a waste of research. The technology you used to make your "its not even wrong" post comes from decades of such research.

There are many scenarios where computers analysing, sorting, manipulating and retreiving information from a photograph is of great benefit to science, especially forensic science. I'm sure you could think of many more applications if you tried. Rubbishing scientific enquiry because you are (a) closed-minded and (b) because the work is (at the moment) basic and used for a somewhat trivial experiment is entirely missing the point.

Maybe they should spend some time creating an algorithm to compress your comments, much more useful. Not everyone will agree with you, deal with it, I do. As a side note, learn NOT to take everything so seriously mate, you'll live a longer happier life that way. ;)

'If successful the algorithm may have serious consequences for the flexible rubber lens hood market, though early indications suggest polarizing filter manufacturers will be safe for some time to come.'

I'm so glad the main concern here is for possible effects on industry and commerce rather than the implications for personal privacy.

I know some people see this stupid. But as an architectural shooter I can picture a number of cases where minimizing reflections in glass on interiors and exteriors would have saved me a lot of retouching.

It's no joke. The music industry, print media, taxis have all been decimated by new technologies. Flexible rubber lens hoods are next. And as goes flexible lens hoods, so goes the entire lens hood market.

Is this not a 'wink' to the fact that a non-petal type hood can be pressed flush to the glass, thus eliminating reflection? Of course, you could do the same with most bare lenses, or an affixed filter.

I just use a multi coated B+W polarizing filter, when turned the right way it eliminates those reflections, alternatively if the picture is very important i use my superpowers to levitate outside the window while i take the shot.

I think this will be good for all those camera phones of which there are tens of millions around the world.

Do you have one for your ultra-wide angle lens with bulbous front element.

The original application was in machine vision. I was working on something similar in eliminating specular reflections to be able to 3D image metals and came across this work 2 years ago. It may of been useful if our project was not canned.

You are assuming this algorithm is just for professional photographers. It is for everyone, including the tourists on trains, planes, and in automobiles who cannot open a window to get the shot of a lifetime. Even pros find themselves in that situation on occasion.

Double reflections were in 96/197 test images -- that's less than 49% that this algorithm has any hope of working on. Still cool that it almost sort-of works with such a computationally cheap algorithm, but would any press coverage have been given this if it wasn't associated with MIT? (Were any previous algorithms?) The article also credits someone at the Hebrew University of Jerusalem as co-developer of the algorithm, yet the article title here says the work comes from MIT. Well, this "news" did come from the "MIT News Office." ;-)

Light reflected from glass is mostly polarised when it is reflected at 45 degrees, but if it's from a window straight in front of you a polariser has very little effect,because the reflected light is barely polarised in this circumstance. Why dont you just try and pull out a polariser and try it right now?

That's right... a polarizer will only eliminate reflection of you're at a certain angle to the window. This seemingly would work in a greater variety of situations. Besides, the technology is really interesting...

To the people here: A polorizer will do the job at 99% of angles including the one in the photo, if not then you bought a CHEAP polorizer.

And Feraudy yes thats why people are unable to take clear images out of airplane windows, oh wait nevermind many have done so at an "impossible" angle with a high QUALITY filter, never skimp out on good glass nor good filters.

In other angles the polarization is weak and no polarizer regardless of quality could effectively remove reflections, unless the light is for some reason polarized to begin with (like coming from LCD monitor or such).

A polarizer will only remove reflections when you look at a glass window under an angle. I believe the optimal angle is about 53 degrees. Your filter won't do much when you take a picture straight through that window.

@ThatCamFanHow could you reply in such a confident manner when you clearly have no clue what you are talking about? My intention is not to insult you, I just can't believe what I sometimes have to read here.. Not everyone's an engineer and there's no need for being so, but why can't you guys just stop posting 'facts' when you don't know what you are talking about?Just google Circular Polarizer and you will see that the 'circulation' has an absolutely different effect.

I tend to skip the comments more and more due to the massive amount of bs, but this was just too much to ignore..

ThatCamFan: unfortunately you are incorrect. If you have ons, I suggest you run a little experiment. It's productive to learn the limitations of any tool you own (although the CPL is a powerful one, it doesn't allow us to bend the laws of physics).

If you have a lightsource directly behind you, a polarizer won't help to remove reflections when you shoot straight through a window. The polarizer being circular has nothing to do with that. Try it.The only thing that helps, beside the MIT software, is pressing the lens against the glass or completely darken the room your in.

A circular and a linear polarizer have exactly the same effect on the image. The only difference is that the light has circular polarisation when leaving the filter and then hitting the sensor. The only reason to use a circular polarizer is if you use an SLR camera.

Agreed. My current workaround is to press the smartphone lens up against the glass. This of course severely limits creative freedom, but eliminates most (off not all) reflections. With a little cropping it can still yield am interesting image sometimes.

Such negative comments! I think this is an interesting development with potential applications outside the field of (hobby) photography. In any case, such research always makes me marvel at the complexity of the human brain. After all, 99.9% of the time, we have no problems distinguishing reflections from the actual scene (and even 'subtracting' them to some extent in our minds).

Currently, mostly vicodin. Nevertheless, as the article itself already states, this might be of use in the field of robotics. If one would have an autonomous robot running through one's house (and a recent visit to the Korea Advanced Institute of Science and Technology's advanced robotics lab has convinced me that this is more fact than fiction), it'd be nice if it could differentiate between the adjacent room and its reflection in the patio doors, for example...

It also has a great application in forensics - from both sides. You can clear away reflections OR isolate them (the whole "zoom... enhance" TV/movie deal, modulo reality), depending on which is interesting. But yes, it is disappointing to see people reading the announcement and thinking to themselves, "cool, I can do the whole 'Ansel Adams' trip without even getting out of the car/bus now!" It ain't all about the shutterbugs.

Laurent Demanet is at MIT and is one of the principle people involved in researching and developing Curvelets. Last I checked, Photoshop's denoising filter was a hard-thresholded reconstruction in a 2D Wavelet basis, which is easier/cheaper computationally. But, for complex images Curvelets can be far superior for denoising (they contain information about angles). They're used in geophysics, etc. The problem is they're more complicated, computationally expensive, and very very patented. I'm sure Adobe and others have looked at these methods, though; there may just not be a business case.

jnd I am not a researcher nor a scientist, your not a politician but you still get to vote who gets into office. See my point here? It does not take a genius to think "cancer cure" rather than "how do I take a dump without having to sit down?"

My guess is that most of the problems you consider meaningful do little to advance technology or society.

It's because of people who solve such "trivial matters" that we can drive cars or type these messages to each other.

Someday this "meaningless exercise" will lead to someone being able to capture cellphone photos or video of a robbery from OUTSIDE the store, through the glass. And all kinds of other useful deeds your mind can't fathom right now.

ThatCamFan, your analogy with voting is similar in a way that choosing candidates once in few years will hardly get you any useful results. If you need to change something to your preferred order, you need to be more active, engage and talk with politicians and local government representatives. As you know, the pre-election promises are quite different from real results after. So with research the same. And the best way is of course to do it yourself. Smart messages in discussion forums will not help us much :)

Latest in-depth reviews

The Leica Q2 is an impressively capable fixed-lens, full-frame camera with a 47MP sensor and a sharp, stabilized 28mm F1.7 Summilux lens. It's styled like a traditional Leica M rangefinder and brings a host of updates to the hugely popular original Leica Q (Typ 116) that was launched in 2015.

The Edelkrone DollyONE is an app-controlled, motorized flat surface camera dolly. The FlexTILT Head 2 is a lightweight head that extends, tilts and pans. They aren't cheap, but when combined these two products provide easy camera mounting, re-positioning and movement either for video work or time lapse photography.

Are you searching for the best image quality in the smallest package? Well, the GR III has a modern 24MP APS-C sensor paired with an incredibly sharp lens and fits into a shirt pocket. But it's not without its caveats, so read our full review to get the low-down on Ricoh's powerful new compact.

The Olympus OM-D E-M1X is the ultimate sports, action and wildlife camera for professional Micro Four Thirds users. However, it can't quite match the level of AF reliability offered by its full frame competitors.

Latest buying guides

What's the best camera for under $500? These entry level cameras should be easy to use, offer good image quality and easily connect with a smartphone for sharing. In this buying guide we've rounded up all the current interchangeable lens cameras costing less than $500 and recommended the best.

What’s the best camera costing over $2000? The best high-end camera costing more than $2000 should have plenty of resolution, exceptional build quality, good 4K video capture and top-notch autofocus for advanced and professional users. In this buying guide we’ve rounded up all the current interchangeable lens cameras costing over $2000 and recommended the best.

What's the best camera for shooting sports and action? Fast continuous shooting, reliable autofocus and great battery life are just three of the most important factors. In this buying guide we've rounded-up several great cameras for shooting sports and action, and recommended the best.

What’s the best camera for less than $1000? The best cameras for under $1000 should have good ergonomics and controls, great image quality and be capture high-quality video. In this buying guide we’ve rounded up all the current interchangeable lens cameras costing under $1000 and recommended the best.

If you're looking for a high-quality camera, you don't need to spend a ton of cash, nor do you need to buy the latest and greatest new product on the market. In our latest buying guide we've selected some cameras that while they're a bit older, still offer a lot of bang for the buck.

We've updated our waterproof camera buying guide with the latest round of rugged compacts, and we've crowned a new winner as the best pick in the category: the Olympus TG-6. That is, unless you happen to find a good deal on the TG-5.

Researchers with the Samsung AI Center in Moscow and the Skolkovo Institute of Science and Technology have created a system that transforms still images into talking portraits with as little as a single image.

K&R Photographics, a camera store in Crescent Springs, Kentucky, was robbed by armed men, who not only took thousands of dollars worth of camera equipment, but also injured the 70-year-old co-owner of the store.

The new Fujifilm GFX 100 boasts some impressive specifications, including 100MP, in-body stabilization and 4K video. But what's it like to shoot with? Senior Editor Barnaby Britton found out on a recent trip to Florence, Italy.

It's here! The long-awaited next-generation Fujifilm GFX has been officially launched. Click through to learn more about the camera that Fujifilm is hoping will shake up the pro photography market - the GFX100.

We've known about the Fujifilm GFX 100 since last fall, but now it's official: this 102MP medium-format monster will be available at the end of June for $10,000. In addition to its incredible resolution, the camera also has in-body IS, a hybrid AF system, 4K video and a removable EVF.

According to DJI, any drone model weighing over 250 grams will have AirSense Automatic Dependent Surveillance-Broadcast (ADS-B) receivers installed to help drone operators know when planes and helicopters are nearby.

Chris and Jordan are kicking off a new segment in which they make feature suggestions to manufacturers for the benefit of all photographer-kind. To start things off, they take a look at the humble USB-C port and everything it could be doing for us.

The Olympus TG-5 is one of our favorite waterproof cameras, and the company today introduced the TG-6, a relatively low-key update. New features include the addition of an anti-reflective coating on the sensor, a higher-res LCD, and more underwater and macro modes.

The Leica Q2 is an impressively capable fixed-lens, full-frame camera with a 47MP sensor and a sharp, stabilized 28mm F1.7 Summilux lens. It's styled like a traditional Leica M rangefinder and brings a host of updates to the hugely popular original Leica Q (Typ 116) that was launched in 2015.

We've been playing around with a prototype of the new Peak Design Travel Tripod and are impressed so far: it's incredibly compact, fast to deploy and stable enough for the heaviest bodies. However, the price may turn some away.