'Fake' World Press Photo isn't fake, is lesson in need for forensic restraint

Paul Hansen's photo as it was submitted to the World Press Photo of the Year AwardPaul Hansen, Sweden, Dagens Nyheter

Update: Neal Krawetz has responded to Hany
Farid's comments in this article. They can be found at the end of
this story.

This year's winner of the World Press Photo of the
Year Award was Swedish photojournalist Paul Hansen's arresting shot
of a group of Palestinian men carrying the bodies of two of their
nephews down a narrow, sunlit alley. The two boys
-- two-year-old Suhaib Hijazi and three-year-old Muhammad
Hijazi -- were killed by an Israeli airstrike on 19 November
2012.

It captures an intense moment of heartache and fury, but many
people -- you might be one of them, reader -- found it a bit...
strange. The lighting is dramatic but unnatural, the facial
expressions so defined as to be almost caricature-like. It's
clearly been manipulated in post-production to make it more
dramatic, but then we shouldn't be surprised at this. Look at the
winning photographs from a decade ago compared to the last two or three years and
there has been a clear move away from realism towards what some
writers have called " illustration-like" styles.

"It's clear that it was touched up, and if you talk to
the jury they say, yes, they know, they have the raw image."

A debate over whether this is a good thing for photojournalism --
especially when it comes to what is perhaps the most prestigious
photojournalism prize -- is definitely needed. Is it fair that the
image might not have won without such dramatic post-processing?
However, the uncanny valley that Hansen's image seemed to fall into
led to one computer scientist calling foul in a very public
way.

Neal
Krawetz, a computer scientist with a specialty in computer
forensics, posted on the Hacker Factor blog that "Hansen's picture
is a composite". He goes on: "This year's 'World Press Photo Award'
wasn't given for a photograph. It was awarded to a digital
composite that was significantly reworked. According to the contest
site, the World Press Photo [WPP] organises the leading
international contest in visual journalism. However, the
modifications made by Hansen fail to adhere to the acceptable
journalism standards used by Reuters, Associated
Press, Getty Images, National Press Photographer's
Association, and other media outlets."

It's not explicitly calling the picture a fake, but it's not far
off. Importantly, the issue revolves around the version of the
picture that won the award, not the original "raw" image. Krawetz's
analysis relied upon not having access to the original, which the
jury for the prize did have, and who repeatedly backed their
decision to award the photo the prize.

Understandably, to nip this in the bud, the World Press Photo
invited two independent experts to forensically analyse the
photograph. The press release, posted up in the evening of 14 May,
crows "digital photography experts confirm the integrity of
Paul Hansen's image files".

The experts referred to were Eduard de Kam from the Nederlands Instituut voor Digitale
Fotografie, and Hany Farid and Kevin Connor of image forensics
and authentication startup Fourandsix. Farid is also a
computer sciences professor at Dartmouth, while Connor worked
previously at Adobe as vice president of product management, with
particular oversight of the Photoshop team.

To get a grasp of just what went on with the WPP controversy,
Wired.co.uk spoke to Farid about his and Connor's analysis:

Had you been aware of the controversy over Hansen's
photo? I'd heard of it. The photo award was given early in the
year, and there was a little bit of concern over the heavy-handed
dodging
and burning, but it sort of went away, then this blog started
making some more serious accusations that in fact it was a
composite of various photos and that's what exploded.

"We told him privately it is wrong, and why it is wrong,
and he kept insisting that he was right. I have no patience for
that, he is impugning the reputation of a photojournalist and
that's wrong."

What was your gut feeling the first time you saw
it?
I mean it's clear that it's -- if I can use this term --
"Instagram-y". It's clear that it was touched up, and if you talk
to the jury they say, yes, they know, they have the raw image. You
can see that it's been locally and globally manipulated. The tone
has been adjusted, and it looked dramatic to me, it looked like the
drama had been done in post, but frankly it's not my business to
tell the WPP what their standards are for photo contests. At the
time they were comfortable with it, and the reason they felt they
had to respond was because the accusations went beyond "this has
been retouched in post", to "this has been a composite of multiple
photographs".

So, you were specifically just looking at the composite
issue?
We were specifically analysing the claims made by Neal Krawetz. He
claimed three things, two of which are just spectacularly wrong,
and one of which is completely inconclusive. The first one was that
he was claiming the image's metadata, specifically the XMP fields,
showed that three files had been combined. That's simply wrong, and
it's wrong because he doesn't understand how metadata works. What
happens is every time you open an image in raw and save it, it
keeps tacking on what has been done to it, and the fact you have
multiple XMP entries is not, as he claimed, evidence of
compositing. It's simply evidence that you opened and closed the
image several times.

Second is he claimed the date in the metadata showed it was
morning. That's incorrect because he doesn't understand basic
geometry. He made these lines in the image to connect shadows to
objects -- that's correct -- and all of the shadows in the image,
intersected a point. That's an analysis developed several years
again to show not where the sun is, but whether the shadows in an
image are consistent with a single light source. It's a
misrepresentation or misunderstanding of the geometry of this.

You know, I don't mind people making mistakes. What I mind is
what I call "doubling down on the dumb". We told [Krawetz]
privately it is wrong, and why it is wrong, and he kept insisting
that he was right. I have no patience for that, he is impugning the
reputation of a photojournalist and that's wrong. Hence my
irritation.

Can you find the sun this way?
No. It's wrong. It's completely ambigious where the sun is. That
intersection point depends on where the photographer is relative to
the sun. What the intersection is is the projection of the light
into the image plane, and that, of course, can be anywhere. I can
make it so it looks below the ground plane by just turning my back
to the sun, but surely you're not going to claim that the light
source is coming from below.

"We understand this in forensic science, and our job is
to quantify those mistakes and be able to say something statistical
and rigorous about the likelihood we made a mistake."

What's the third mistake?
The third was this thing Neal developed called Error Level Analysis.
It reveals whether parts of an image have been compressed compared
to other parts. The problem is, first of all, this doesn't give you
an answer, it just gives you an image, and this image is completely
indecipherable. It incorrectly labels altered images as original
and incorrectly labels original images as altered with the same
likelihood.

He has this very handwaving notion of what this is supposed to
do which I don't understand, and I've been doing this for a long
time and I'm reasonably good at it. I don't think it's interesting
or reliable, or quantifiable or quantitative, or particularly
scientifically grounded. When we do analyses, we know what our
error rates are. We know the chance of making a mistake is X
because there are always mistakes. We understand this in forensic
science, and our job is to quantify those mistakes and be able to
say something statistical and rigorous about the likelihood we made
a mistake. Neal's mistake is he doesn't understand some basic facts
about metadata and geometry, but that's not his biggest mistake. He
has a level of certainty that is unwarranted, and that's dangerous
in forensic science. In the excitement of the blogosphere and
Twitter people get crazy.

Taking that level of uncertainty into account, how sure
are you that you can you say this is a genuine
photo?
The good news is we have the raw image. This is stored in a
proprietary format that is not impossible to reverse-engineer, but
it's incredibly difficult to do. We had the metadata associated
with that which told us what he did in raw and we could compare
that with the final image and do a side-by-side comparison, and we
could say this is clearly not a composite of multiple photographs
taken at different times spliced together. That is a patently false
claim we can say with a very high degree of certainty.

Everything after that is an editorial question. He clearly did
global and local manipulation of colour and tone. The jury knew
what he did, they had the raw images, they compared it and said
"you know what, we're comfortable with this". There is something
interesting about this, it seems like a lot of these juries are
moving towards this "Instagramming" of photos, this idea that
there's this drama you can create with this heavy photo retouching
in post. Is that a good idea? I don't know, I think that's a
question for photojournalists and people who do this for a
living.

But with a movement towards this Instagramming of
photos, aren't competitions inviting this kind of
controversy?
I think it's really interesting. The fact is, at a time where
people are particularly sceptical and particulalry critical of the
media, maybe they should be a little bit more careful on the degree
of retouching they allow. It's outside of my area of expertise, but
I think that it's the most interesting dialogue that came out of
this. I've talked to a few reporters about this, and they're
thinking "yes, we shouldn't have accepted this". Let's have that
conversation, but let's also not pretend that this guy did
something he didn't do.

----

As an epilogue, it's worth pointing out that Krawetz has updated his blog to reflect further analysis he's conducted,
and says that he feels "vindicated" in his initial suspicions after
doing his own side-by-side comparison of the raw, original photo
and the version that won the WPP Award. He specifically says that
he "never said that people were pasted into the
street", but "speculated that multiple pictures could have
been combined" -- so, other images were pasted in, just not in an
obvious way (example: marks on the heads of the children and the
uncles carrying them).

His update instead focuses more on the question of "which
picture is the most authentic?", which goes back to the subjective,
editorial decision to allow touching up in photojournalism. That's
a debate that's still taking place.

***

Neal Krawetz responds to Hany Farid's comments
below

Dear Wired,

A few days ago I discovered your 16 May interview with Dr. Hany
Farid concerning the World Press Photo award controversy. This
letter addresses several inaccuracies presented by Dr. Farid.

"[Krawetz] was claiming the image's metadata, specifically
the XMP fields, showed that three files had been combined. That's
simply wrong, and it's wrong because he doesn't understand how
metadata works."

According to Adobe's XMP specifications, the "Document Ancestor" field
denotes "a copy-and-paste or place operation". There are four
Document Ancestor records in the photo that I evaluated. These
explicitly identify four documents that were combined with the base
image. Combining pictures to create an image is the definition of a
"composition".

On my own web site, I have a detailed evaluation of the XMP metadata. This
evaluation shows that there were at least five pictures combined in
three stages to form the final composite image. This metadata does
not identify what was in each of these combined pictures or how
they were combined. The metadata only shows that the combining
occurred.

"[Krawetz] claimed the date in the metadata showed it was
morning. That's incorrect because he doesn't understand basic
geometry."

The metadata in the Gaza photo does not contain any geometric
information. However, it does contain a timestamp that says
2012-11-20 at 9:39:38+01:00. The time, 9:39am in GMT+01:00, is
10:39am in Gaza (GMT+02:00). The last time I checked, 10:39am was
considered "morning".

"He made these lines in the image to connect shadows to
objects -- that's correct -- and all of the shadows in the image,
intersected a point. That's an analysis developed several years
again to show not where the sun is, but whether the shadows in an
image are consistent with a single light source. It's a
misrepresentation or misunderstanding of the geometry of
this."

The convergence test validated that the shadows are consistent,
but the facial lighting is inconsistent.

"We told [Krawetz] privately it is wrong, and why it is
wrong, and he kept insisting that he was right."

I had no private communications with Dr. Farid. I had an email
exchange with his business partner, Kevin Connor, on 13 May. The
main focus of the email exchange was on the XMP data and not on the
shadow/lighting analysis. In the final email, Connor wrote:

No, I'm afraid you're mistaken about this metadata. You will
*not* see this happen if you open a new/different raw file. The
portion of the metadata you're looking at doesn't communicate any
information whatsoever related to potential compositing.

This misunderstanding of the XMP metadata comes from ignoring
the unique document IDs and Document Ancestor fields that identify
compositions, as detailed in my XMP analysis. Connor sent me two sample
images that he claimed proved his point. The samples failed to
replicate the metadata seen in the file that I analysed and did not
counter my argument that XMP metadata can specifically identify the
type of composition used in this image.

"The third was this thing Neal developed called Error Level
Analysis. ... The problem is, first of all, this doesn't give you
an answer, it just gives you an image ..."

The Error Level Analysis (ELA) algorithm measures the
compression level seen across the image. It provides an answer to
the question: "What is the JPEG lossy compression potential across
the picture?" It generates an image because that is the clearest
way to show the results.

"[ELA] incorrectly labels altered images as original and
incorrectly labels original images as altered with the same
likelihood."

ELA does not label images as original or altered; it only
identifies the compression rate's lossy potential. It is up to the
user to interpret the results. Any errors in identification rest
solely on the viewer. Nearly a million users have accessed the FotoForensics website, where
they can use an ELA system, and not one of them has seen the system
automatically generate conclusions about an image's
authenticity.

The scientific method is to observe, hypothesize, predict, test,
verify, and validate. We can observe that the quality of a JPEG
image degrades each time it is saved. We hypothesize that
subsequent resaves will result in a quality level that does not
increase. We predict that the next resave will lower the quality of
the JPEG. We test by resaving the JPEG again and comparing the
amount of change. We verify by noting that the quality did not
increase, and we validate that the hypothesis agrees with the
observation.

From this basis, we can extend the findings using the scientific
method. For example, if an image is modified, then the modified
sections can have a different error level potential. Other
researchers have extended upon ELA with their ownpositiveresults.

In very technical terms, ELA quantifies the gradient descent of
the discrete cosine transform due to JPEG's integer truncation. The
results are repeatable and consistent.

"We know the chance of making a mistake is X because there
are always mistakes. We understand this in forensic science, and
our job is to quantify those mistakes and be able to say something
statistical and rigorous about the likelihood we made a
mistake."

Confidence intervals are part of a statistical
hypothesis. ELA uses a numerical hypothesis, not a statistical
hypothesis.

Algorithms that quantify compression, noise, and other image
attributes operate like a microscope. They highlight artifacts that
would otherwise be invisible. What are the false-positive and
false-negative rates for a microscope? It's a trick question; a
microscope does not have them. As with other non-statistical
systems, a microscope only identifies artifacts. It is up to a
human to identify possible scenarios that are consistent with the
observations. We observe the test results, we hypothesize possible
causes, we predict, test, verify, and validate.

Even if you do not understand the analysis or the tools, it is
simple enough to verify the results:

-- My hypothesis was that the file was a composite. I predicted
that multiple files were combined to form a composition. This was
tested by referencing the metadata, which identifies multiple files
that were combined. It was validated through an interview, where the photographer Paul Hansen
explicitly stated that he combined multiple pictures to enhance the
image.

-- I identified areas that appeared edited. Farid's own review
identified "global and local" modifications. Although he
identified no "significant" photo manipulation, he did not say that
there was no photo manipulation.

As we have seen, altered images result in emotional responses.
Sometimes, however, the most telling part comes from what is not in
the image.