If this is your first visit, be sure to
check out the FAQ by clicking the
link above. You may have to register
before you can post: click the register link above to proceed. To start viewing messages,
select the forum that you want to visit from the selection below.

Validity of the Psychopathy Checklist

Despite recent evidence that scores on the Psychopathy Checklist-Revised (PCL-R) vary widely in adversarial legal contexts depending on which party retained the evaluator, the test has become increasingly popular in forensic work. In Texas, indeed, Sexually Violent Predator (SVP) evaluators are required by statute to measure psychopathy; almost all use this test. It is not surprising that prosecutors find the PCL-R particularly attractive: Evidence of high psychopathy has a powerfully prejudicial impact on jurors deciding whether a capital case defendant or a convicted sex offender is at high risk for bad conduct in the future.

But a current effort by the instrument's author, Robert Hare, to suppress publication of a critical article in a leading scientific journal may paradoxically reduce the credibility of the construct of psychopathy in forensic contexts.

That's the opinion of two psychology-law leaders, psychologist Norman Poythress and attorney John Petrila of the University of South Florida (two authors of a leading forensic psychology text, Psychological Evaluations for the Courts), in a critical analysis of Dr. Hare's threat to sue the journal Psychological Assessment. The contested article, Is Criminal Behavior a Central Component of Psychopathy? Conceptual Directions for Resolving the Debate, is authored by prominent scholars Jennifer Skeem of UC Irvine and David Cooke of Glasgow University. The study remains unpublished.

"[T]he threat of litigation constitutes a serious threat to academic freedom and potentially to scientific progress," write Poythress and Petrila in the current issue of the International Journal of Forensic Mental Health. "Academic freedom rests on the premise that advances in science can only occur if scholars are permitted to pursue free competition among ideas. This assumes that scholars have the liberty to do their work free from limitations imposed by political or religious pressure or by economic reprisals."

According to Poythress and Petrila, after the critical article passed the peer-review process and was accepted for publication, Dr. Hare's lawyer sent a letter to the authors and the journal stating that Dr. Hare and his company would "have no choice but to seek financial damages from your publication and from the authors of the article, as well as a public retraction of the article" if it was published. The letter claimed that Skeem and Cooke's paper was "fraught with misrepresentations and other problems and a completely inaccurate summary of what amounts to [Hare's] life's work" and "deliberately fabricated or altered quotes of Dr. Hare, and substantially altered the sense of what Dr. Hare said in his previous publications."

In general, defamation claims must prove that a defendant made a false and defamatory statement that harmed the plaintiff's reputation. Truth is an absolute defense. Critical opinions are also protected from defamation actions, as are "fair comments" on matters of public interest.

In this case, the contents of Skeem and Cooke's contested article have not been made public. However, it is hard to see how critical analysis of a construct that is enjoying such unprecedented popularity and real-world impact would NOT be of public interest.

Poythress and Petrila express concern that defamation claims against opposing researchers, while traditionally rare, may be becoming more common, leading to a potentially chilling effect on both individual researchers and the broader scientific community. Like so-called SLAPPS -- Strategic Lawsuits Against Public Participation -- used by corporations and other special interest groups to impede public participation, even meritless defamation lawsuits extract heavy penalties in terms of lost time and money and emotional distress.

Judges have been critical of pretextual deployment of defamation lawsuits, Poythress and Petrila report; a judge in one case warned that "plaintiffs cannot, simply by filing suit and crying 'character assassination!,' silence those who hold divergent views, no matter how adverse those views may be to plaintiffs' interests. Scientific controversies must be settled by the methods of science rather than by the methods of litigation."

Researchers avoid conducting critical research out of fear of lawsuits.

Academics decline to serve as volunteer peer reviewers for academic journals due to loss of anonymity in defamation suits.

Journal editors self-censor on controversial topics.

As Poythress and Petrila conclude:

Because publication of the article by Professors Skeem and Cooke has effectively been long delayed, if not ultimately suppressed, one clear impact of this threat to sue is that researchers who may have been willing to investigate alternative models of psychopathy that might have been derived from the Skeem and Cooke article are not able to do so, simply because the article is unavailable. Because science progresses, in part, both by confirming viable models and disconfirming nonviable ones, the suppression of information relevant to constructing candidate models for empirical evaluation can be viewed as impeding the progress of science….

It seems clear from our review that such threats strike at the heart of the peer review process, may have a chilling effect on the values at the core of academic freedom, and may potentially impede the scientific testing of various theories, models and products. In our view it is far better to debate such matters in peer review journals rather than cut off debate through threats of litigation.

In court, meanwhile, the effects of Dr. Hare's threat may prove paradoxical. Attorneys whose clients could be prejudiced by introduction of the Psychopathy Checklist may be able to discredit the instrument by pointing to the suppression of critical literature about the underlying construct of psychopathy.

The full article is PCL-R Psychopathy: Threats to Sue, Peer Review, and Potential Implications for Science and Law. A Commentary, by Norman Poythress and John P. Petrila, in the current issue of the International Journal of Forensic Mental Health. The abstract is available HERE; the full article requires a subscription.

Related:

"The Dark Side of Peer Review," by Stephen D. Hart, also in the current issue of the International Journal of Forensic Mental Health (abstract HERE

Re: Questions raised about the validity of the Psychopathy Checklist

I have been arguing pretty much the same thing for years. The PCL does not measure psychopathy. It attempts to do so but fails. What it measures is criminal propensity, not the same thing at all but something probably more closely resembling the criteria for the DSM-IV-TR Antisocial Personality Disorder diagnosis.

The problem in essence is this:

The PCL-R breaks down into two factors. Factor 1, which attempts to measure the "core personality features" of psychopathy (basically along the lines of Cleckley's criteria); and Factor 2, which measures criminal behavior and criminal personality.

Factor 1 is a weak factor in terms of predictive validity, interrater reliability, etc. Thus, the part of the PCL which tries to measure the essence of psychopathy is pretty much a failure.

Factor 2, which is heavily influenced by historical data regarding past antisocial and criminal behavior, is much stronger, both in terms of predicting future criminal behavior (not surprisingly, given that all things being equal the best predictor of future behavior is past behavior) and in terms of interrater reliability.

This is not exhaustive: There are other problems with the PCL-R. But this is the fundamental problem with the scale in a nutshell.

Conclusion: The PCL-R is a measure of antisocial/criminal potential and/or "criminal personality". It is decidedly NOT a very good measure of psychopathy. Scoring high on the PCL-R does NOT make an individual a psychopath.

Now, of course Dr. Hare is going to object to the publication of additional data pointing this out. He has made his reputation on the PCL-R and, additionally, has a financial interest in its propagation and popularity.

Re: Questions raised about the validity of the Psychopathy Checklist

Robert Hare is a psychologist who studies psychopaths and is best known for developing the "Hare Psychopathy Checklist" or PCL-R, a standard diagnostic tool for assessing offenders. He is currently threatening to sue two psychologists who wrote an article critical of the theory underlying the checklist, as well as the academic journal, Psychological Assessment, that accepted the piece for publication after it was peer-reviewed.

There's an account of the affair over at the excellent forensic psychology blog, In the News, who notes that the article was authored by respected researchers Jennifer Skeem and David Cooke and was titled Is Criminal Behavior a Central Component of Psychopathy? Conceptual Directions for Resolving the Debate. As a result of the legal threat the article has never come to light.

The letter from Hare's lawyers apparently claimed that the he would:

"have no choice but to seek financial damages from your publication and from the authors of the article, as well as a public retraction of the article" if it was published. The letter claimed that Skeem and Cooke's paper was "fraught with misrepresentations and other problems and a completely inaccurate summary of what amounts to [Hare's] life's work" and "deliberately fabricated or altered quotes of Dr. Hare, and substantially altered the sense of what Dr. Hare said in his previous publications."

It's probably worth noting that the PCL-R is big business. At current prices, each assessor who uses the checklist needs their own copy of the manual ($123) and the rating booklet ($68.50) and each individual assessment requires an interview guide at $5 each and a scoring form at about $3 each.

Because the assessments are used in the legal system, it is important that no-one (like an opposing lawyer in court) can find fault in the process and attending the 'official' training from the PCL-R company is considered the gold standard.

Recently, the affair has caught the attention of two lawyers and legal scholars who have just published their own analysis of the situation in the International Journal of Forensic Mental Health.

They express regret that Hare has chosen to use legal threats to counter his critics rather than to refute any points he felt were unfair in print himself, but also note that his strategy may actually undermine the usefulness of the PCL-R in court as opposing lawyers "may attempt to discredit that testimony by arguing that the literature relevant to evaluating the PCL-R has been tainted".

Overwhelmingly, opinion is that Dr. Hare shot himself in the foot by threatening legal action against the researchers and the journal. Hopefully, this debacle will serve as a cautionary tale for others whose research undergoes critical scrutiny due to forensic or other public-policy implications.

Re: Questions raised about the validity of the Psychopathy Checklist

Hi Dr Baxter,

Assuming that your critique of the PCL-R is correct (I am familiar with the PCL-R, but not the literature critiquing it) and that the PCL-R does not measure psychopathy very well, my question to you is what does measure psychopathy well?

I appreciate that the PCL factor 2 list is a reasonable approximation of antisocial personality disorder, and that its useful and predictive of reoffending. I'm sure I've read somewhere that Hare claims that there are three personality types measured by his scale:

High Factor 1's only: The successful psychopath, who is selfish and callous but behaviourally controlled
High Factor 2's only: The antisocial individual, who is not actually psychopathic, but may be criminal and will likely meet the criteria for ASPD.
High in both factors: Criminal psychopath, most likely incarcerated.

I appreciate that other researchers have suggested a 3 factor analysis of the PCL - suggesting selfishness, narcissism and criminality.

Just to repeat though, if factor 1 is invalid or lacks inter-rater reliability, what does measure psychopathy?

Also, might factor 1 lack validity precisely because it is a personality scale, and psychopaths tend to lie about such things? In contrast, factor 2 is about lifestyle, for which the evidence can't be denied.

Re: Questions raised about the validity of the Psychopathy Checklist

Originally Posted by Tom Michael

Assuming that your critique of the PCL-R is correct (I am familiar with the PCL-R, but not the literature critiquing it) and that the PCL-R does not measure psychopathy very well, my question to you is what does measure psychopathy well?

I don't know of a psychometric test which even comes close. That was why there was initial excitement and enthusiasm about the PCL (similarly to the initial enthusiasm and susbsequent disappointment with the MCMI for personality disorders in general). It would be nice to have a valid instrument to "diagnose" psychopathy. To date, we don't have one.

Originally Posted by Tom Michael

I appreciate that the PCL factor 2 list is a reasonable approximation of antisocial personality disorder, and that its useful and predictive of reoffending. I'm sure I've read somewhere that Hare claims that there are three personality types measured by his scale:

High Factor 1's only: The successful psychopath, who is selfish and callous but behaviourally controlled
High Factor 2's only: The antisocial individual, who is not actually psychopathic, but may be criminal and will likely meet the criteria for ASPD.
High in both factors: Criminal psychopath, most likely incarcerated.

That's all rather circular, though. First, the validity of Factor 1 is, as I've stated, poor, so high Factor 1 scores don't really mean anything at all, or they may mean any number of other things besides psychopathy. Second, Factor 2 does measure criminality - not very surprising since it is based on the presence or absence of a history of criminal antisocial acts. But really what does it tell us that we don't already know from looking at criminal record?

Originally Posted by Tom Michael

Also, might factor 1 lack validity precisely because it is a personality scale, and psychopaths tend to lie about such things? In contrast, factor 2 is about lifestyle, for which the evidence can't be denied.

I don't think lying is the problem - that's a problem for any scale and any population, but it is possible to construct scales with (a) validity scales to detect lying and (b) low transparency or face validity. Additionally, the PCL is a rating scale rather than an objective or self-report test.

Rather, the problem with the PCL, especially Factor 1, is that it's a poorly constructed scale with high face validity but poor interrater reliability and poor predictive validity.

Re: Questions raised about the validity of the Psychopathy Checklist

Thanks for your reply David. My work is mainly to do with frontal lobe brain injury, but I have an interest in psychopathy due to the similarities between orbitofrontal brain injury and ASPD, and early amygdala injury and psychopathy. What I have initially liked about the PCL is that it did appear to distinguish those two types.

If the PCL has low inter-rater reliability then it does fail as a measure, although that could be one reason that the company that market it insist on training the PCL raters (lack of experience is one reason for poor inter-rater reliability). I guess my second question to you would be this; If we lack any valid psychometric test or measure, can we really be sure that psychopathy exists?

I'm reasonably certain that it does, as there are physiological markers, such as low skin conductance responses to violent images, and decreased frontal activity in brain imaging. Also, there do exist individuals who seem to be totally self centred and who have a callous disregard for others. These individuals will generally lie and make out that they are sincere and caring, although I'm sure I've read this in work by Hare and others, and it does tend to make one a little paranoid, seeing psychopaths everywhere.

Can we really only detect psychopaths in a qualitative manner? Is there any theoretical obstacle to producing a valid rating scale? or just practical barriers such as the person being rated lying? I appreciate there are tests of malingering for psychometric tests, but that is a different type of deception than pretending you are a caring and sincere person.

Re: Questions raised about the validity of the Psychopathy Checklist

I would not say that the failure of the PCL-R is any reason to assume that psychometric measures of psychopathy are not possible or that psychopathy does not exist. I have met a few classic psychopaths (i.e., Cleckely's criteria) in my career and they are unmistakable - like lokking into the eyes of a shark.

But psychopathy does not equate to criminality. Not all psychopaths have a criminal rfecord and not all criminals are psychopaths (indeed, most criminals are not). I think using Cleckley's criteria is a good starting point but creating a psychometric test from that basis is obviously not as easy as it looks. I personally don't think that a 3 point rating scale founded on a subjective estimate or judgement of whether an individual has or does not have a specific characteristic is the way to go.

Add to that the problem of diagnosing via cut-off (e.g., if the PCL-R score is equal to or greater than 30, he is a psychopath; if his score is 29, he is not a psychopath), and the lack of agreement even on what that cut-off should be (the research literature changes the cut-off seemingly on a whim to create better group sizes), and the fundamental concept of the PCL-R seems unsound to me.

As for training, I can tell you that the PCL-R is in widespread use in many venues, especially correctional facilities, parole offices, and the like, where "training" generally involves reading the manual and some of the research literature.

Re: Questions raised about the validity of the Psychopathy Checklist

So then, if we agree that psychopathy is a valid construct, and that they can be identified, is there any way we can quantitively measure their traits with reliability? I'm sure that you have correctly identified those that you have met, but I guess that this must have been a qualitative process, using your knowledge and Cleckley's criteria.

I think the dirty secret of a lot of psychometric measures (as opposed to tests) is that whether we assign a 0, 1 or 2 to a trait on a 3 point Likert scale effectively IS a qualitative judgement. I use the Dysexecutive questionnaire (a measure of symptoms of frontal lobe/dysexecutive brain injury), which has good inter-rater reliability amongst psychologists, but I know from personal experience that this reliability drops off dramatically if you as family members to use it to rate their relatives brain injury symptoms.

What I think this means is that creating a psychopath trait measure is possible, but that unless the raters are well trained, its reliability is questionable. You mention that the PCL is used by many non psychologists, which could be one reason why inter-rater reliability is so poor. I strongly suspect that the factor 1 poor reliability is because it is rating internal personality traits, which can be hard to judge objectively as a third party unless you know that person very well.

What criteria might you replace factor 1 with, in order to form a more reliable list? Might you be in a position to test and validate a new checklist, for use only by clinical or forensic psychologists?

Re: Questions raised about the validity of the Psychopathy Checklist

What I think this means is that creating a psychopath trait measure is possible, but that unless the raters are well trained, its reliability is questionable.

I don't think a rating scale is the way I would go at all. I'd go with something a lot less subjective, and a lot less transparent, along the lines of the MMPI.

You mention that the PCL is used by many non psychologists, which could be one reason why inter-rater reliability is so poor.

No. I said that in my experience it is used by a lot of psychologists who purchase the test and read the manual but who do not have the training. I would add that I'm not sure how much difference training would make in any case. Ultimately, this kind of test is highly subject to rater bias.

I strongly suspect that the factor 1 poor reliability is because it is rating internal personality traits, which can be hard to judge objectively as a third party unless you know that person very well.

And see above: If you believe John Doe to be a psychopath, your ratings are biased in that direction before you even begin to complete a PCL.