Sunday, January 11, 2009

Dr. Itiel Dror on Confirmation Bias

Dr. Dror contributed an article to Criminal Investigative Failures. The following is an article by him on confirmation bias with forensic evidence (which is supposed to be bias proof). Not so fast says Dror.

The article originally appeared in the Summer issue of the UK's Police Review (apologies for the formatting errors):

Biased brains

The way police officers and staff examine evidence is always at risk ofbeing far from impartial. Itiel Dror investigates how unconscious humanbias can inadvertently influence how scientific evidence is interpreted

How is it possible that highly skilled andprofessional police forensic expertsmake mistakes?

Examination of evidence by forensic examin-ers, investigations by detectives, or consideringwhich action to take as police constables, allhave one thing in common: they rely heavily onhuman thought processes and the brain.Thought processes and perception are farfrom perfect because the way people internallyunderstand, interpret, evaluate and judge infor-mation highly depends on how thought proc-esses are structured and brain mechanisms. Toooften people overemphasise the role of the in-formation itself and neglect to understand thecrucial role that the human mind plays in under-standing and interpreting this information.People’s brains process information, but theydo not have the resources and capacity to dealwith all the information they receive. Thereforethey have evolved to take ‘short cuts’.This means prioritising and selectively ex-amining information, actively and dynamicallyprocessing, and other mechanisms that formthe basis of intelligence. As people becomemore experienced and highly skilled, they in-creasingly develop and rely on these short cuts.Examining one piece of evidence is used toguide the search and processing of further in-formation, piece after piece, in a way that theyall fit together to solve a puzzle. Knowing whereto look, what questions to ask, paying attentionto the important things and knowing where tofind them, is what distinguishes experts fromnovices.

However, as one piece of information guidespeople’s search and evaluation of subsequentinformation, so they can also be led astray.Once people have a belief or a hunch of whatthe data may suggest, a theory or hypothesis,this has powerful and profound effects on howthey perceive it, the way they process the in-formation and the mental representations theyform of this data, how they evaluate and inter-pret this information, and their judgements anddecision making.

Diminished objectivity

These effects can take many different forms andinfluence people in a variety of ways. For exam-ple, confirmation bias is when people notice andgive more weight to information that is consist-ent and supports certain interpretations and notothers. Conversely, people do not notice, dis-miss, or give less weight to other informationthat does not fit (or even contradicts) the in-terpretations they unconsciously support. Con-firmation bias is only one example of the waypeople think that diminish experts’ objectivity.Escalation of commitment and momentum,conformity and group think, prophecies thatfulfil themselves and wishful thinking are justa few other psychological and cognitive phe-nomena where experts unavoidably and uncon-sciously can lose objectivity and be selective andbiased.

Myself and my research team of David Charl-ton, Ailsa Peron, Ina Schmitz-Williams and PeterFraser-Mackenzie set up to experimentally ex-amine effects of context on forensic experts. Ina series of studies undertaken over several yearswe provided forensic evidence and examinedwhether its evaluation by forensic experts wassolely based on the evidence itself.

For example, we would present fingerprintsfrom a crime scene, and observed if the conclu-sion by forensic experts on whether they matchdepended on if the suspect confessed to thecrime. We consistently found that such contex-tual information affected the judgement anddecisions made by qualified and experiencedforensic examiners.

In a couple of these studies we presentedidentical fingerprints to the same fingerprint ex-perts, but provided a different external contextfor them each time. We found that the context in which evidence is presented, such as thatdescribed above, can cause the same forensicexaminer to reach conflicting decisions on iden-tical evidence. Our data and research findingsfrom these studies suggest that such influencesare most powerful when the quantity and qual-ity of the evidence is low, and that these effectsoccur at a subconscious level without the foren-sic examiner being aware of them.

Practical evidence

Are confirmation bias and thought process influ-ences an academic issue existing purely withinthe theories of the human mind and brain?Well, try to say this to Brandon Mayfield, anOregon attorney who was arrested for killing191 people and injuring more than 1,800 in theMarch 2004 Madrid train bombings.Based on a latent fingerprint left at the crimescene by the real Madrid bomber, OuhnaneDaoud, FBI fingerprint experts positively identifiedMr Mayfield as the bomber. Even an independ-ent forensic expert appointed to his defence teamconcluded that it is a definite match.

In May that year, after Daoud was identified asthe owner of the fingerprints, the FBI acknowl-edged the error and partly attributed it to confir-mation bias. Mr Mayfield was released from cus-tody and has since recieved an apology from thefederal goverment and was awared USD2 millionin compensation.

The issue of questionable objectivity and biaswhen examining evidence within a leading con-text is not limited to fingerprints or to investiga-tions in the US.

For instance, CCTV evidence was recently usedin the Old Bailey in the case of Levi Bellfield whowas convicted of the murders of Amelie Dela-grange in 2004 and Marsha McDonnell in 2003in southwest London. He was also found guilty ofthe attempted murder of Kate Sheedy in 2004.One piece of evidence in in the attemptedmurder charge relied on a CCTV image of a car. However, there was only a single frame from theCCTV footage that contained the registrationnumber, and this was of extremely low quality.Initial examination of the image by detectives(with minimal context) was able to conclude verylittle information about the number plate.However, when the image was presented toforensic experts along with a suspect’s registra-tion plate (for example, that of the accused), thenthe forensic imagery examination of the CCTVimage was conducted within a potentially influ-encing and biasing context.

The Met was eventually forced to admit incourt that detectives had failed to properly exam-ine the CCTV footage and four officers were laterformally reprimanded by the force after a reviewby the Independent Police Complaints Commis-sion.

Possible solutions

I believe these examples show that there is noquestion that forensic experts and police offic-ers (like everyone else and like experts in otherdomains) are susceptible to bias and other influ-ences. So what can be done about this? The solu-tions to this problem, both in the forensic domainas well as in the larger context of policing, is two-fold.

The first solution is the development and im-plementation of best practice in the field. Anexample of this is for forensic experts to try toexamine evidence without potentially biasing in-formation being given to them.

Best practice needs to be scientifically basedand validated by experts in thought processesand not by forensic experts.

When this is not possible (which does happendue to operational requirements), the aim shouldbe to first examine the evidence without the con-text, clearly documenting the more objective andindependent analysis, and only then to allow theintroduction of the additional contextual and po-tentially biasing information.

Contextually biasing influences come in manydifferent forms. Another example of such contextand how it can be managed is the proactive stepstaken by Kevin Kershaw, the head of forensicservices at Greater Manchester Police who is cur-rently on secondment to the National Policing Im-provement Agency to work on these issues. He isactively working to combat this bias and protectshis forensic examiners from being unduly influ-enced by buffering them from the investigatingdetectives.

The second solution is training. There is gener-ally an alarming lack of training in this area. Forexample, in the Levi Bellfield case, both CCTV im-agery forensic experts stated under cross-exami-nation in court that they acknowledge the exist-ence of confirmation bias, but had no training inthis area.

The Fingerprint Society, Hampshire Constabu-lary, and Greater Manchester Police are examplesof a professional body and forces who have pro-vided some training in this area.

Forensic evidence is an integral and importantpart of policing and the criminal justice system. Itis relied on more and more, and it is vital to makesure that this, as with other police processes anddecision making, is as professional and objectiveas possible.

Understanding the human brain and mind,and the structure of thought processes, is vitalto ensure the highest quality in judgement andperformance.

Dr Itiel Dror is a senior lecturer in cognitiveneuroscience at the University of Southamp-ton. He has worked with the US Air Forceand police forces across the world

1 Comments:

I recently came across your blog and have been reading along. I thought I would leave my first comment. I don't know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.