Cylab.info

Accred Qual Assur (2003) 8:179–183DOI 10.1007/s00769-003-0609-9
D. Brynn HibbertAbstract An Australian case studyKeywords Forensic science ·
Received: 23 October 2002Accepted: 17 February 2003
of abuse were acquitted on appealbecause of shortcomings in the prosecution’s case that establishedthe identity of the material seized.The need to have proper standard operating procedures that can be followed routinely and correctly ishighlighted.
2052 Sydney, NSW, Australiae-mail: [email protected] au
A recent judgment in the Court of Criminal Appeal of
In this paper the analytical chemistry of the case is
New South Wales highlights a problem faced by courts
explained, and broader issues are discussed that relate to
and scientists [1]. Australia has an adversarial system of
matters relevant to accreditation and quality assurance.
justice that is derived from the English system. In each
The nature of expert opinion in the Australian system is
case, a plaintiff has to persuade the court of his or her
case, while the defendant argues the opposite. The end ofthe case is when the Court, a judge acting as a jury, or ajury of ordinary men and women, determine for oragainst the plaintiff’s case. The case in hand is the trialof Regina (the Queen) v three gentlemen who werefound guilty of ‘knowingly taking part in the manufac-ture of not less than the commercial quantity of a prohib-ited drug, namely methcathinone.’ On appeal, the con-victions were overturned on the grounds that the jury didnot take sufficient account of the shortcomings of theCrown’s evidence of identity of the alleged drug.
In the local press, much was made of the suggestion
that they were really trying to make the male anti-impo-tence drug ‘Viagra’ (sildenafil)! If this were indeed a se-
rious defence it is difficult to see how a synthetic chem-
Structures of 1-[[3-(6,7-dihydro-1-methyl-7-oxo-3-propyl-
1H-pyrazolo [4,3-d]pyrimidin-5-yl)-4-ethoxyphenyl] sulfonyl]-4-
ist might have decided to make structure (1) via structure
methylpiperazine monocitrate (Sildenafil marketed as Viagra) and
(2) (Fig. 1).
(2) methcathinone
It is easy for scientists to believe that lawyers cannot ap-
preciate the proper scientific method, but in this case thetrial judge stated the problem with some clarity [4].
The case discussed here highlights the importance of fo-rensic evidence being of the highest quality, and being
– . gave in evidence that, in relation to these tests, he
able to be shown to be so in court. The issue was not
had not done them in accordance with the manual –
complex, and was essentially: was a quantity of white
“technically”, again he said – in that he had not run a
powder found in the premises of the defendants the
standard at the end of each blank, nor had he included
chemical substance methcathinone (Fig. 1 (structure 2))
duplicate runs as indicated by the manual. He claims,
however, that the reference to “duplicates” relates to
The initial analysis was by gas chromatography mass
quantitation, not identity testing, even though the
spectrometry (GCMS), identifying methcathinone against
manual apparently indicates otherwise. He agreed that
a literature spectrum (in the absence of a reference sam-
his runs in respect of these samples, however, was
ple). The GC retention time was later compared with that
“technically” deficient, because they did not comply
of a reference standard of methcathinone. The transcript
of the trial refers to a ‘manual’, which appears to have
You have heard the evidence relating to the impor-
been the standard operating procedure (SOP), which
tance of complying with these set standards and coun-
specified the degree of matching of peak intensities be-
sel’s submissions to you in relation to that . You
tween the mass spectrum of the unknown and a literature
may ask yourselves whether substantial non-compli-
reference spectrum (±20%). Under cross examination the
ance with these set standards, designed to produce
analyst admitted that the practice of the laboratory was
validity in the results of an analysis, can be properly
to perform visual matching only, and that in fact the dif-
ferences of peak heights between the sample spectrumand a literature spectrum were indeed greater than 20%.
Those interested in quality assurance in the chemical
There is no doubt that the laboratory should have fol-
laboratory may thank the learned judge for pointing out
lowed the manual. However, it may be seen why the ar-
what the laboratory may have temporarily forgotten.
bitrary ±20% was ignored. First the barrister read the cri-
When a standard was finally sourced, the attempts of the
terion “±20% of reference spectra” as an instruction to
laboratory to match the retention time with that of the
take the peak height of the reference and determine lim-
seized material were also riddled with error. Given (from
its at ±20% of that height. In this case the most abundant
where it is not clear) a window of ±0.2 min to match the
ion intensity was of m/e 58, and measured against this
GC retention time of a sample with a standard, the one
intensity the next peaks in the literature were m/e 77
chromatogram presented to the court showing a match
(11%) and m/e 51 (11%). These literature values are giv-
had a retention time of 9.55 min compared with the stan-
en in whole number percentages which implies at least a
dard 9.8 min. Under cross examination it was admitted
range of ±0.5% before any other considerations of preci-
that the window might have been increased without doc-
sion are made. In cross examination the barrister deter-
umentation or reason, in what must be said was unpro-
mined that the allowable ±20% range of m/e 51 and 77
was ±2.2%. As 1% of this is already taken up in the
It is clear to me that the scientists were convinced that
significant figures, it may not be surprising that it is dif-
they had identified methcathinone, but fell into the trap
ficult to find a match between the ion intensities mea-
of believing that their own belief was all that was needed
sured in the forensic laboratory and those in the literature
to satisfy the court. The fact that the jury found the de-
taken under possibly very different conditions. The mea-
fendants guilty shows that, despite the shortcomings of
sured ion intensities of m/e 58 and m/e 77 were 29% the evidence, they believed the laboratory’s story. It wasand 24% respectively, and much was made of the dis-
left to the judges of appeal to understand the need for ab-
solute and demonstrable reliability of the chemical evi-
It was argued by the witness from the forensic labora-
tory that the manual was ‘not supposed to be read in a
The importance of quality assurance in forensic evi-
black and white fashion’ [2] which merely compounded
dence has been highlighted by a judge of the New South
the problem, faced with the interpretation by the defence
– “5.28 It is essential, in those circumstances, that there
– It’s a part of the quality accreditation re laboratory.
be an added emphasis on the adoption of high stan-
You must have adequate documentation of the proce-
dards of competence, adherence by laboratories to
dures that you use for analytical chemistry, and these
protocols, and acceptance of a duty to the court on the
procedures and protocols are to be followed by all
part of criminalists and forensic scientists, as an ad-
junct to the adoption by the Courts themselves of pro-
cedures, which are designed to improve the quality
and the manner of presentation of expert evidence.
5.29 In this regard the need for quality assurance
To clarify the role of an expert witness, and to provide
and quality control is imperative, carrying with it
guidance for experts and lawyers, the Federal and State
matters such as the regular checking of forensic labo-
courts of Australia have adopted similar codes of con-
ratories by random blind sampling, the recording of
error rates, and the giving of attention to the training
The Federal guidelines start with a clear definition of
of technicians not only in a way which ensures their
competence, but which also brings home to them the significance and potential consequences of their
work, so far as suspects and victims of crime are con-
– An expert witness has an overriding duty to assist
the Court on matters relevant to the expert’s area ofexpertise
An expert is, at the court’s discretion, absolved from thehearsay and opinion rules and so must take care that he
The saying ‘to be wise after the event’ may well be ap-
or she does not abuse this freedom. The guidelines then
plied to this paper. However, the essence of good quality
explain how an expert should establish his or her exper-
assurance systems is that they promote being wise before
tise, how to set out evidence, and how to document prop-
the event, thus avoiding the unfortunate consequences of
erly the exact nature of the expertise on which the evi-
dence is based. Finally the expert must add the following
The problem in this case was the difference between
the analyst’s absolute certainty about the identity of thesample, and his inability to demonstrate this to the court.
– “[The expert] has made all the inquiries which [the
While professional judgement should count for a great
expert] believes are desirable and appropriate and that
deal, the courts demand solid and very formal proof.
no matters of significance which [the expert] regards
Therefore members of a forensic laboratory should:
as relevant have, to [the expert’s] knowledge, beenwithheld from the Court.”
1. Be well aware of the standard operating procedures
and other systems underpinning the accreditation of
The need for this declaration has helped experts stand up
to clients who, while not wishing to suborn a witness in-
2. Expect to follow those procedures until a satisfactory
to actively lying, have suggested there is no need to
mention everything that has come to light (especially the
3. Be able to explain in court the scientific basis of their
bits that are prejudicial to the client’s case). I would
argue that this approach has an advantage over practice
4. Not to allow themselves to be pressured by manage-
in the USA courts, where the expert is very much part of
ment, police or judicial officers to produce a substan-
In the case at hand, one of the problems identified by
the Appeal Judges was the very poor quality of the evi-
This case also highlights the need to involve everyone in
dence given by the defendants’ expert analytical chemist.
continual rounds of quality improvements. The matching
On the witness stand he railed against the work done by
requirements using library spectra clearly do not work. A
the forensic laboratory in quite intemperate language.
simplistic ±20%, although sounding plausible, was not
Despite correctly pointing out the obvious deficiencies in
suitable for this situation. It may be that a technical per-
the identification of the alleged drug, he allowed himself
son in the laboratory had noticed this problem before,
to overstate his case to the point where the jury may
but without the culture of preventative action he or she
have stopped listening to the science. Perhaps the fol-
would not have been encouraged to raise the matter.
lowing taken from the judgment, as an example of the
Changing SOPs is not an easy matter once a laboratory
language used by this expert, could well be used in train-
has been accredited, but it can be done if properly docu-
ing expert witnesses how not to conduct themselves in
mented and justified. The effort would have been well
Accreditation to ISO/IEC 17025 [6, 7] should ensure
– Some of the results he described as “absolutely amaz-
that the proper infrastructure, methods, staff and proce-
ing”. He described the GCMS results of August 2000
dures are in place, and it will be of interest to see what
as “.this rubbish proffered in evidence.”. He de-
steps are taken by the Australian accreditation body
scribed test results as “dreadful” and obliquely ac-
cused the Crown experts of being biased.
The jury were in the best position to assess his de-
will never be able to convict a drunk driver again. Law-
meanour and these illustrations show why they might
yers have also advanced that opinion, and barristers have
not have been inclined to place much weight on his
occasionally taken the opportunity to pretend ignorance
of even the most basic statistics. In a patent case inwhich I had measured blood glucose levels by two meth-ods on each of eight replicate samples, and had shown
by a t-test that the means of each method differed signifi-cantly, I was asked the following. “Professor, you have
In the case presented here, neither expert came out with
made 16 measurements?” “Yes”. “And they are all dif-
any credit. The forensic laboratory failed to maintain its
ferent?” “Yes”. “So surely, 15 of them must be wrong!”
standards, did not produce reliable results and so the
I forbore from correcting him – “at least 15 of them are
prosecution eventually failed. In particular the evidence
given in court by the prosecution’s analyst appeared to
Against this pessimistic view of the courts’ ability to
suggest that he was not fully prepared nor did he under-
embrace uncertainty comes discussions (unpublished) in
stand the importance of the admissions he was forced to
the Australian Academy of Forensic Science, in which
make under cross examination. The defence expert had
senior judges have pointed out that although the ultimate
the easier task of explaining to the court the nature of
judgment (Guilty/Not Guilty) has no uncertainty, to ar-
standard operating procedures and their place in a quali-
rive at that point the court has to weigh many matters
ty system, and why the disregard shown by the forensic
that are uncertain. In assessing the credibility of witness-
analyst rendered the identification of the substance unre-
es, the weight a certain piece of evidence is to be given,
liable. However he also did not manage to do his job
and so on, a judge or jury must grapple with impondera-
properly, because the jury placed no weight on his evi-
bles. Scientific evidence is of a better quality because
dence, possibly because of the manner in which he gave
quantifying uncertainty places bounds on results that are
usually absent from other types of evidence. There may
It is important that expert witnesses understand that
be a conception that scientific evidence has to be abso-
their job is to provide the court with appropriate opinion,
lute, otherwise the very credibility of ‘Science’ is at
and not to try the case in lieu of the judge and jury. It
stake, but this is to make too much of the scientist’s evi-
may sometimes be difficult. In a recent case, I discov-
dence, which must take its place with all other aspects of
ered that an expert called by the opposition had used
the case. A better dialog with the courts should allow un-
atomic numbers instead of atomic weights to determine
derstanding of an uncertainty expressed with a given
the amount of substance from a measured weight. In
confidence interval. It may be that a court will want
conference with the barrister, I explained that this
measurements made to a greater precision than the con-
schoolboy howler would allow him to repudiate totally
ventional 95% confidence interval, which, of course, can
the witness’ evidence. However the barrister remained
unimpressed, reminding me that I had explained that anatomic number is about half the atomic weight of an ele-ment and so the relative values calculated would not
change that much. Despite my entreaties that this wassuch an egregious fault that the so-called expert could
Grappling with statistics is not, as we have seen, a strong
not have survived the ignominy of its revelation in court,
point of the legal profession. The concept of probability
the barrister did not make the point. He had realised that
is crucial to assessing the worth of DNA matching [12],
there was enough against the opinion of this witness, and
or indeed the matching of any chemical data. More com-
that the correction would not have made a deal of differ-
plex issues have the power to confuse, if handled in such
ence to the discussion of the results, while confusing the
a way. A famous case of illogical reasoning was during
court still further over the nature of atomic chemistry.
the trial of the American personality O J Simpson, in
The barrister’s job is to win the case, not to strike a blow
which a statistician advised that the probability of a
known wife beater actually killing his wife was verysmall (1/10,000). Writing it as a conditional probabilityPr(M|B)=0.0001 where M is a murdered wife, and we
are given the prior information that the husband is a wifebeater (B). As pointed out in a letter to Nature after the
There is an opinion that courts cannot accept the concept
trial [13], the real probability that was of significance is
of uncertainty. It is said by some forensic scientists that
Pr(G|M,B) – the probability that given a murdered wife
if evidence is presented on, for example, blood alcohol
(M), and the fact that the husband was a wife beater (B),
levels, with a certain (low) probability that the concen-
that the husband was the killer (G). This can be estimat-
tration was actually below the legal limit [10], courts
ed by a Bayesian approach to be at least one third.
Courts in England [14] and Australia [15] have set
surance, SOPs and the like, is to bring demonstrable
their faces against the overuse of statistics to give what
rigour to a complex procedure. Courts bring out the
appears to be a definitive probability of the guilt or inno-
worst in the system. Clever lawyers can tease out in-
cence of the defendant. The case of R v Adams estab-
consistencies of approach or transgressions of SOPs.
lished that to use Bayes’ Theorem or similar method in a
When they are desperate, the system allows for consid-
criminal trial to bring together probabilities “plunges the
erable obfuscation in cross examination. However this
jury into inappropriate and unnecessary realms of theory,
is not the fault of the legal system. When liberty is at
and complexity deflecting them from their proper task”
stake mere opinion, no matter how well founded, is no
substitute for evidence correctly acquired, interpretedand defended in court. If, as may be the case here, thelaboratory was accredited to follow SOPs that were not
appropriate, the way forward is not to disregard theSOPs, thus invalidating the whole process, but to seek
An analyst has a primary duty to determine properly
what he or she is asked to do, whether for court, or any
There is clearly plenty of scope for education, of legal
other client. The existence of accreditation, quality as-
9. Jasanoff S (1995) Science at the bar.
quirements for accreditation in the field
tion of Testing Authorities, Sydney,Australia
expert witnesses in proceedings in the Federal Court of Australia. Federal Court Practice direction, Federal Court of Australiahttp://www.fedcourt.gov.au/pracproc/practice_direct.html

GLOSSARY • Conjuncture: • Adhocracy: The actual balance of unevenly developed social and politi-The term was coined by Alvin Toffler. According to him, thiscal forces in a society at a particular moment is referred tosystem would replace bureaucracy in which several tran-sient project teams or task forces would do the work which • Counter Factual: would be dissolved after com