Wednesday, February 13, 2008

Tonight, I'm going to write myself an Aston Martin

This is the story of my attempt to 'cheat' in an on-line spot-the-ball competition to win an Aston Martin. It's also the story of my failure, but you get free source code that implements automatic detection of image alteration using copy/paste or tools like the Clone Tool in Photoshop.

First, take a look at this photo:

Notice anything strange? In fact this image has been tampered with to cover up a truck. The truck is completely hidden by foliage. Here's the original:

Wouldn't it be nice to be able to detect that automatically? It is possible. Here's an image automatically generated by my code showing what was moved. All of the red was moved to the blue (or the other way around).

I was motivated to work on this program by greed (or at least my never-ending love of having a little flutter on things). Best of the Best runs spot-the-ball competitions in airports to win very expensive cars. But they also run the same competition online. That meant I could get my hands on the actual image used... could I process it to discover where the ball had been removed? (In reality, this isn't the right way to win because the actual ball position is not governed by where it actually was, but where a judge thinks it was).

Would it be cheating if I could? Apparently not, the competition rules say I should use my skill and judgment in determining the ball position. Surely, skill covers my programming ability.

Unfortunately, I know nothing about image processing. Fortunately, the paper is written in a very clear style and a bit of Internet research enabled me to track down the knowledge I didn't have. (Also, thanks to Jessica for sending me the original images she used to test my implementation).

In brief the algorithm does the following:

Slide a 16x16 block across the entire image from left hand corner to bottom right hand corner. For each 16x16 block perform a discrete cosine transform (DCT) on it and then quantize the 16x16 block using an expanded version of the standard JPEG quantization matrix.

Each quantized DCT transformed block is stored in a matrix with one row per (x,y) position in the original image (the (x,y) being the upper left hand corner of the 16x16 block being examined).

The resulting matrix is lexicographically sorted and then rows that match in the matrix are identified. For each pair of matching rows (x1,y1) and (x2,y2) the shift vector (x1-x2,y1-y2) (normalized by swapping if necessary so that the first value is +ve) is computed and for each shift vector a count is kept of the number of times it is seen.

Finally the shift vectors with a count > some threshold are examined, the corresponding pair of positions in the image are found and the 16x16 blocks they represent are highlighted.

Here's another picture showing a golfing image that's been touched up to remove something from the grass:

To get access to image data I used the FreeImage library and wrote a small C program that implements Jessica's algorithm. You can download the source here; it's released to you under the GNU GPL.

The program has two key parameters that affect how the image is processed: the quality factor and the threshold.

The quality factor is a number used to 'blur' the image (actually it changes the quantization): the higher the factor the more blurring and hence more 16x16 blocks are likely to seem the same to the algorithm. Increasing the quality factor will tend to increase the false matches.

The threshold is simply the number of blocks that have to appear to have been copied together. This prevents us from seeing a single 16x16 block as evidence of copying. Increasing the threshold means ever larger groups of blocks have to be identified together before they are identified as copying.

Back at Best of the Best I grabbed the image for Supercar Competition (SC-272), cut out a section that I thought the ball had to be in (just to speed up processing) and ran the algorithm. After some parameter tweaking the algorithm came up only with what look like false matches to me (along the bar where it's all one color):

And, of course, that's not where the judge thought the ball was. So, I guess I won't be driving home in the V8 Vantage, but what geek needs that when they've got a cool piece of software that detects copy/move forgery in images?

Which leaves me with one question: how are spot-the-ball images generated? Is this an algorithm problem, a problem because they use JPG (which is already transformed) for their images, or are these images generated in some other way?

With a little bit of work, I've got it compiling on MSVC++ 2003, and it works great.

The inner loop during "Building DCT transformed matrix" is a bit slow. I was able to get a huge speedup by precalculating all the pixel values for the image beforehand (i.e. the values of variable "pixel"), thus eliminating a lot of calls to FreeImage_GetScanLine() and round(), which are quite expensive.

Altogether it's a very effective algorithm! I've been very impressed by the results.

For the record, the main changes required for MSVC are casting the void* values to the appropriate pointer types, and also implementing round(x) as floor(x+0.5).

I'm guessing they create the spot-the-ball photos by taking two photos, offset in time, with a static camera. Then a careful copy - paste with a soft edge from one image to the other to remove the ball. That wouldn't show up as a clone operation.

Recent versions of Photoshop have built-in tools to automatically merge multiple images. You load up a series of images and mask / mark areas to keep or remove in each, and it blends it all into one "seamless" image. It's a handy trick to get a clean image of something when you're in a crowd and people keep getting in your shot, for example.

Here's a free tool from MS to do it: http://research.microsoft.com/projects/GroupShot/or http://www.snapmania.com/info/en/trm/index.html

Darren You have said that you were able to run it on VS2003I am using VS2005 over the dotnet framework 2.0/3.5 and this piece of code is driving me nuts.I do confess that my C is very rusty i have migrated to c# 6 years go. i am facing problems with the "stat" function as it produces a compile time error stating that there is no overload that takes 2 parameters! whats crazy is that intellisence saais it needs to parameters so does MSDN. If it is not too much trouble help me out on this or better yet uplod the code you were able to run of MSVS.thank you

hello i am undergraduate student working on this paper for my final semester project. can you please give me Matlab code for this paper, that will help me to complete my engineering. will be waiting for for kind response.

ok thank you for your kind response , can you please tell me how to get the images on which you have tested your c code, i mean the images which fridrich has gives to you ,kindly if you can mail them to me at [email protected]

thank you for your kind response, you have done a good job. i will be working on to translate your c code into matlab, if a find difficulty in understanding your code i will seek your guidance, thank you regards , ghiayas

hello , i have implemented this algorithm in matlab till the point where lexicographical sorting is done. can you please explain the next point in more detail , where rows of matrix A(lexicographically sorted matrix) are compared and shift vector counter was used , i can not understand that point.

Hi John,Thanks for the method on detecting manipulated images(copy-paste).I have a question regarding on lexicographically sorted.can you help me about lexicographically sorted.is it possible to give me an example?.so, i can implement in matlab.Thanks & Regards,prd johnny

Hi John,Thanks for the method on detecting manipulated images(copy-paste).I have a question regarding on lexicographically sorted.can you help me about lexicographically sorted.is it possible to give me an example?.so, i can implement in matlab.Thanks & Regards,prd johnny

Hi John,Thanks for the method on detecting manipulated images(copy-paste).I have a question regarding on lexicographically sorted.can you help me about lexicographically sorted.is it possible to give me an example?.so, i can implement in matlab.Thanks & Regards,prd johnny

FYI: I put your code into a GIMP plugin and added multithreading. Also I replaced some of the C arrays with C++ vectors, which makes it more stable handling bigger images.Same URL:https://sites.google.com/site/elsamuko/forensics/clone-detection

Hi, thanks for your great tools. Actually several papers are based on this concept, they just change the features to change the approach (DCT, wavelet coefficient, SVD,...)

However I ran your code, the output are 2 .png images and they're just images with 1 red strip at the top and 1 blue strip at the bottom. The rest are just black. I couldn't get the result like the given image here. What does it mean? I try to read the source code but maybe it's related to some routine of the 3rd party lib.

hello John Graham-Cumming plzz. can you help my my project is in copy move forgery detection using dwt the steps are

first - Algorithm for Detection of Reference and Match Blocks:

1. Read the image selected by the user as input.2. If the input image is not a gray scale image thenconvert it into a gray scale image.3. Apply wavelet transform up to specified level ‘L’ tothe gray image.4. For each overlapping b × b block in the LLL image4.1. Form a matrix A of dimension b2 columns and(M-b+1) × (N-b+1) rows by extracting theresulting pixel values by rows into a row of A.4.2. Form another matrix B same as A with twoadditional columns for storing top-left coordinates.5. End6. Ignore blocks where contrast is minimum.7. Sort matrix A lexicographically.8. For each row of A8.1.Compute the phase correlation for the blockcorresponding to the current row with the blockscorresponding to ‘p’ rows above and below thecurrent row.8.2. If the computed maximum phase correlation valueexceeds a preset threshold value‘t’, then store thetop left coordinates of the corresponding referenceblock and the matching block from B matrix in anew row of a matrix.9. Endsecond: Algorithm for Comparison of Reference and MatchingBlocks:

1. For LLL-1 level in the image pyramid1.1. For each row of the matrix1.1.1. Form a reference region by padding ‘m’pixels on all the sides of the b × breference block.1.1.2. Form a matching region by padding ‘m’pixels on all the sides of the b × b matchingblock.1.1.3. For each b × b overlapping of the referenceregion.1.1.3.1. Find corresponding match inmatching region based on Phasecorrelation but search process has tobe opted for selected part of matchingregion.1.1.3.2. If the computed maximum phasecorrelation value exceeds a presetthreshold value, then the top leftcoordinates of the correspondingreference block and the matchingblock are stored in a new row of amatrix.1.2. End2. End3. For LLL-2 level to original image in the image pyramid3.1. For each row of the matrix3.1.1. Form a reference region by padding ‘m’pixels on all the sides of the b × b referenceblock.3.1.2. Form a matching region by padding ‘m’pixels on all the sides of the b × b matchingblock.3.1.3. Compare them using Phase Correlation.3.1.4. If the computed maximum phase correlationvalue exceeds a preset threshold value, thenstore the top left coordinates of thecorresponding reference block and thematching block in a new row of a matrix.3.2. End4. End5. Plot the blocks as copied and pasted regions on thegiven input image

i can do this step (Ignore blocks where contrast is minimum.)

We remove all blocks with lowcontrast. We tell that a block haslow contrast if the difference betweenits maximum intensity pixel and itsminimum intensity pixel is lower thansome predefined threshold T.This helps to prevent noisy results, suchas 2 identical blue patches of the sky

how can I choose the optimal threshold T to preform this step plzz help me.