Adobe Photoshop content-aware filling technology

What if the spot healing brush could be made smart enough to know where to automatically sample data from when repairing an image?

When the healing brush and patch tool first made their appearance in Photoshop 7 they generated a storm of interest. Here at last was a way to make retouching easy. At a stroke the healing brush could seamlessly blend the source data with the destination. The healing brush and patch tool were on their own a pretty good start, but we later saw the introduction of the spot healing brush (first in Photoshop Elements and later in Photoshop) which would automatically select the source point for you.

So in case you have never used the spot healing brush before, I should briefly explain that when using the spot healing brush there is no need to select a source. You simply click with the spot healing brush on a blemish and Photoshop automatically works out the best place to sample from and removes it. While this works well the majority of the time, you can’t expect the technology to be smart enough to always know precisely where best to sample from. In some instances Photoshop will sample incorrectly – especially if you retouch too close to a sharp edge – and you will either have to undo and take another stab at using the spot healing brush or use the normal healing brush or clone stamp tool instead.

All that may change one day in the future. What if the spot healing brush could be made smart enough to know where to automatically sample data from when repairing an image?

Adobe researchers Eli Shechtman and Dan Goldman, working together with Prof. Adam Finkelstein from Princeton University and PhD student Connelly Barnes, have introduced PatchMatch, which is a “randomized correspondence algorithm for quickly finding approximate nearest neighbour matches between image patches”.

This technology is also now referred to as “content-aware filling” and here is my understanding of how it works.

When random samples are used to estimate how to repair an area, in most instances the random samples will be wrong. But if a random search covers a wide enough area, there is an increased chance that a few lucky guesses will manage to locate a portion of the image that offers an almost exact match. Plus in places where this happens, the sample areas immediately surrounding that particular point may also prove to be good candidates for use in repairing a section of the image.

So in use, this new method of healing carries out a wide, random scan of the surrounding area and where the search locates an area that looks like it will be a good match for repairing the section of the image you are painting, will concentrate on sampling pixels from there and the surrounding area to produce an improved guess when painting to mend parts of an image.

To give you an idea of how this new technology might one day be incorporated into Adobe Photoshop, you’ll find an interesting story on John Nack’s blog: http://blogs.adobe.com/jnack (see links below).

Plus the examples provided below will also give you an indication of what will one day be possible in Photoshop.

In the first example there were a lot of cables criss-crossing the landscape that spoiled the view. These were retouched out using a clone tool that utilises the new “content-aware filling” algorithm.

In the Figure 2 example you can see the effectiveness of content-aware filling when applied to an image selection. In this example the fill may not have been 100% perfect, but it does a really good job of automatically (and quickly) removing objects from a tricky backdrop.

If you check out the articles on John Nack’s website and the movie link below you’ll have a better idea of how well this technology can work and get a glimpse of some of the further improvements that may also make their way into Photoshop.

Figure 1. Here is a before and after example where the PatchMatch technology was used to easily remove the cables from a landscape scene.

Figure 2. Here is another before and after example, this time showing you how a selection fill was used to quickly remove the horse from this scene and accurately fill the removed area.