Three University of Virginia law professors are questioning the expert evidence behind one of the most controversial cases before the U.S. Supreme Court this term, Dukes v. Wal-Mart.

The court recently heard oral arguments on whether up to 1.5 million current and former female employees can pursue a class action sexual discrimination lawsuit against Wal-Mart, the nation’s largest private employer.

An expert testifying for the plaintiffs examined Wal-Mart’s employee policies and concluded that the company used personnel practices that were biased against female employees and led to disparities in pay and promotion between men and women. He relied exclusively on the prior scholarship of Law School Professors John Monahan and Larry Walker to justify the method he used to formulate his opinion.

But in a series of recent articles, Monahan, Walker and Professor Greg Mitchell have disavowed the method used by the plaintiffs’ expert and argue that he has misused Monahan and Walker’s theories to justify his testimony.

“The expert evidence provided by the plaintiffs was unreliable and an insufficient basis for the class certification determination,” the professors said in a joint statement.

As authors of the leading casebook on the use of social science evidence in the courts, Monahan and Walker have written a series of articles discussing how courts should evaluate and use social science evidence.

In these articles, they advocate for the use of general social science evidence to provide background information or context that may assist the judge or jury when evaluating case-specific evidence, and they refer to this use of social science research as “social framework” evidence.

“We originally argued that judges should craft jury instructions that inform the jurors on propositions drawn from reliable social science research, and then jurors could decide whether and how to apply these social science propositions to the facts of the case presented at trial,” Walker said.

But Walker and Monahan also stipulated that general social science research cannot be directly applied to the facts of a given case by an expert witness, because the facts of any given case may differ significantly from the conditions under which the research was conducted. Such research only provides probabilistic or general accounts of behavior — not ironclad “laws” about how all people act across all situations, the professors said.

“It’s not scientifically justifiable to assume that any given set of parties acted in accordance with the participants in any given laboratory or field experiment,” Monahan said. “General social science research can only provide general information that lay persons may not have or that may contradict lay persons’ intuitions or assumptions about human behavior.”

For instance, jurors might assume that eyewitness confidence is a strong indicator of accuracy when social science research shows that such confidence is an unreliable or weak indicator of accuracy.

“Under our original social framework proposal, in a case where eyewitness evidence is important to the case, the parties should propose social-science-based instructions on the conditions under which eyewitness identifications are more and less reliable and the judge should give such instructions when convinced the underlying research is reliable and could help the jurors evaluate the evidence admitted at trial,” Walker said.

Although the idea of social framework evidence became increasingly accepted among courts, Monahan and Walker’s proposal for providing this evidence in the form of jury instructions was not widely accepted. Instead, a number of courts allowed experts in some field of social science to inform the jury about research findings that could provide helpful background information for jurors’ consideration of the case-specific evidence.

“A number of experts took their role a crucial step further: They not only summarized general social science research for the judge or jury, they applied the research to the facts of the case as they understood the facts and told the fact-finder how they should interpret the case-specific evidence in light of this social science research,” Monahan said.

Experts who took this extra step of applying the general social science research to the facts of the case began referring to their method as “social framework analysis” and almost uniformly invoked the scholarship of Monahan and Walker as authority for this methodology.

A number of courts, including the lower courts in the Dukes v. Wal-Mart case, accepted the experts’ assertions that the work of Monahan and Walker showed social framework analysis to be an acceptable expert method — even though this method was not endorsed by the professors and was contrary to their understanding of the proper role of general social science evidence in court.

The “social framework” concept has been increasingly used in expert evidence at the class certification phase to establish that class members were suffering from a common harm, such as common policy of discrimination as alleged in the Wal-Mart case. In large-scale class actions, statistical and social science experts are often relied on to present much of the evidence in support of, and in opposition to, class certification, which is not decided by a full trial.

“Experts who perform a social framework analysis of a defendant company’s policies and practices have become increasingly common and often supply the only evidence of a common source of injury to proposed class of plaintiffs,” Walker said.

“This approach is not a method recognized by social scientists as a reliable means for reaching descriptive or causal conclusions regarding what a company’s practices are and whether those practices cause harm to any individuals involved in the case,” Mitchell said.

Monahan, Walker and Mitchell’s most recent scholarship on social framework evidence has not been embraced by the plaintiffs’ bar or legal academics who support the plaintiffs in the Wal-Mart case, but it was welcomed by the defense bar and Wal-Mart. Wal-Mart cited the 2008 article in its merits brief filed with the Supreme Court and urged the court to adopt the professors’ view that the expert evidence provided by the plaintiffs was unreliable and an insufficient basis for class certification determinations.

“Whether the Supreme Court will reach this issue remains to be seen, and some court observers have suggested that there are a host of other issues that may be sufficient to resolve the case,” Mitchell said. “We hope the court will address this issue and restore integrity to the social framework concept as originally conceived.”

Such a ruling would benefit both future plaintiffs and defendants, the professors argue, because it would encourage the use of sound empirical methods to analyze the facts of a case and would prevent future abuses by expert witnesses.

The professors explained their position in a question-and-answer session.

Why do you believe social framework analysis is an inappropriate method for expert testimony?

We find social framework analysis fundamentally flawed in two ways: The data are insufficient and the means of analyzing this data are insufficient. First, relying on a set of documents assembled in litigation, where both parties have an interest in making their case rather than accurately and fully describing an organization, its members and their relations as the basis for making descriptive and causal claims about this organization and its relations with its members violates basic norms of data collection and sampling. Second, relying on private, subjective judgments as the “method” for analyzing this flawed set of data violates basic norms of data analysis. Experts who apply general social science research to an organization make no measurements and conduct no tests to determine whether conditions and relations in the defendant organization are the same as those in the general research they cite. They simply assert that organization is or is not engaging in some harmful practice without any scientific basis for those assertions.

The authority for the opinions of experts using social framework analysis comes down to “Trust me, I’m an expert,” rather than “Trust me, I applied sound scientific principles to reliable data from this case to reach my conclusions.” Social science, and science in general, requires that public, replicable methods be used to formulate opinions, which is never the case in social framework analysis.

What would be a sound method for evaluating Wal-Mart’s policies and practices for evidence of discrimination?

As we discuss in an article forthcoming in the Emory Law Journal (“Beyond Context: Social Facts as Case-Specific Evidence”), experts can use reliable social scientific methods to analyze case-specific data. They do it all the time across a wide range of cases that we discuss in this paper. For instance, social scientists conduct surveys that are regularly used in trademark disputes, and observational studies and surveys are being increasingly used in wage and hour disputes.

In a case like the Wal-Mart case, many different approaches could be used to gather case-specific evidence using social scientific methods. Organizational data can be properly sampled and analyzed to reach reliable conclusions about the practices of an organization, and experiments can even be used within the organization to reach conclusions about the effectiveness of its anti-discrimination policies and practices. Doing reliable case-specific research may be more expensive and time-consuming in some cases, but the rules of evidence don’t create an exception that permits experts to use unreliable methods where using reliable methods would be harder or more expensive.

What should be the foundational principles of expert testimony in civil litigation?

The Supreme Court has endorsed some simple, but wise, rules for evaluating expert evidence. The easiest principle to apply is to ask whether the expert has used methods for her work in court that are as rigorous as the methods that would be used by the expert were the work performed outside court. If the expert is using less rigorous methods in court, that is a sign of potential unreliability that should prompt serious concern on the part of the court. Another foundational principle is the requirement that something more than subjective belief be the basis for an expert’s opinions. Courts should be very wary of experts who cannot disclose some objective means of analysis that another expert could take and apply to the same data. Absent some independent means to evaluate an expert’s opinions, courts would be left trusting the good faith and intentions of the expert to give valid opinions.

What are the dangers of low-quality expert testimony?

In many cases, experts testify to matters about which jurors have little knowledge, and jurors have few tools to evaluate the validity of the expert’s testimony. In such cases, the jury may defer to the expert, and may even see admission of the expert’s testimony as endorsement by the court. It is therefore essential that the court fulfill its gatekeeping role under Daubert v. Merrell Dow Pharmaceuticals to ensure that expert testimony has a reliable basis. Where both parties can provide expert testimony, the adversarial process may help to expose invalid expert opinions, but this will often not be the case, and, in any event, why should we risk the confusion associated with a battle of experts where one expert is offering invalid and misleading testimony?

What role should the American Sociological Association and other professional academic organizations play in this debate?

We think organizations such as the American Sociological Association and American Psychological Association could play a very positive role in promoting sound science in court, but to do so they will have to look beyond the interests of any particular party in a dispute and ask what can social scientists who are members of their organizations properly do in the context of litigation. There is a great temptation to support the cause or interest being represented by one side in litigation, and this immediate interest may be at odds with the long-term interest these organizations have in promoting the use of sound social science evidence in courts.

For instance, the American Sociological Association recently sought to defend the work of the plaintiffs’ expert in the Wal-Mart case in an amicus brief filed with the Supreme Court, but, as we discuss in a forthcoming paper (“The ASA’s Missed Opportunity to Promote Sound Science in Court”), the ASA’s brief fails to engage the actual opinions and method used by the expert. As a result, the ASA ignores serious problems with this expert’s testimony and may encourage other sociologists to use similarly flawed methods when giving expert testimony.

The members of the ASA, as social scientists, should find that problematic, but if that argument seems idealistic, consider this: When in the future members of the ASA confront social framework analysis being used to support some cause that they oppose, they will have sacrificed their principled opposition to the expert’s methodology so long as they excuse social framework analysis used to support causes they support.