reflections on experiments and ethics following WGAPE

WGAPE Inaugural National Meeting, UC Berkeley, May 24, 2012. Photo taken by Lisa Chen, shared with permission from CEGA who reserve the rights to the photo.

It’s been almost two weeks since the inaugural national WGAPE meeting at UC Berkeley. The papers are posted here, but perhaps the one that sparked the most intense dialogue — the paper by Gwyneth McClendon entitled, “Co–ethnicity and Democratic Governance, An Experiment with South African Politicians”– is not linked on the WGAPE page but is available here. Below is the paper’s abstract:

In this paper, I discuss a field experiment in which 1294 South African local politicians were each contacted by a fictitious constituent whose name signaled either co-ethnicity with the politician or out-group membership. The constituent raised a concern either about a public goods issue that is prioritized equally across South African ethnic groups or about one that is more ethnically divisive. I found that politicians of all ethnic groups were more likely to respond to co-ethnic constituents than to non-ethnic constituents and were much more likely to respond to an ethnically unifying issue than to a ethnically divisive issue. These results hold regardless of the expressed partisanship of the constituent, the degree of ethnic heterogeneity of the politician’s district, the degree of competitiveness of the district, and the electoral rules (first-past-the-post or proportional representation) used to elect the politician. The findings have potential implications for our theories linking ethnic diversity to low levels of public goods provision; they also raise new questions about the “homestyle” of politicians in ethnically divided democracies.

Like experiments done in the US, Gwyneth studies the behavior of politicians responding to fictitious constituents. The primary issue among some WGAPE-ers was the use of deception in Gwyneth’s experiment. Some social scientists believe deception is unjustified under any circumstance because it is unethical and damages the reputation of scientists in the larger society (see Baumrind 1985). One WGAPE-er said to me during a coffee break that if reviewing Gwyneth’s manuscript, s/he would reject it and s/he strongly believed it should not be published (I asked if the US study that used the same method should not have been published in one of the top political science journals, to which s/he responded yes and that Leonard Wantchekon’s experiment in Benin was also unethical and should not have been published). The participant’s opinion is consistent with the deception hardliners, who reject manuscripts out of hand if they use deception (I believe this is still the rule in economics). Rick Wilson, a political scientist and leader in laboratory experimentation, does not encourage the use of deception, but also thinks the hardline stance is unreasoned. He wrote in 2007:

I find the position on deception given by most experimental economists to be one of religious belief and moral certitude, rather than a reasoned position. Subjects coming into a lab are already suspicious of what the experimenter is going to do. Most of our experiments are seemingly abstract and leave plenty of room for doubt to creep into the mind of a subject about what we’re doing. Instead, my own position has to do with the reason we pay subjects (see below). We should never deceive subjects about the link between their actions, the actions of others and their payment. If we do, then we’ve lost control over induced value for preferences. If a subject does not believe that their actions are mapped into promised payments, then it is impossible to draw inferences about their preferences. Beyond this, deception can be useful, especially if it is easy for the subject to guess the experimenter’s hypothesis or if it is crucial for ensuring internal validity. Despite this statement deception should be avoided whenever possible. A good experimenter can always find ways to avoid deception and should strive to do so. I will not summarily recommend that a paper be rejected if the experimenter can justify the use of deception and the subject’s link between actions and payments is not broken.

Deception is not uncommon in social science experiments, including the budding experimental literature in political science. A 2006 article in the leading political science journal (the APSR) found that 31% of laboratory experiments published in the APSR used deception. It is particularly common in psychology experiments. Eric Dickson discusses deception at length in a chapter of a handbook on experimentation in political science (previous, ungated version of book here), where he concludes:

These examples suggest that deception may offer access to certain research questions that would remain inaccessible in its absence.

In a newly published issue of The Experimental Political Scientist, Gwyneth wrote a piece devoted to the ethics of using public officials as experimental subjects. In it, she points out the many studies that have used deception, particularly those that are topically related to her own. She also describes in more detail her original design, which called for debriefing the subjects (and advising them of the deception) — but her institution required that she removed the debriefing protocol before allowing her to do the research. She puts conditions on the use of deception, and feels in particular that scientists could only use deception if they themselves would bear the brunt of any associated negative feedback.

Perhaps because of my training, I’ve taken the stance that I’d like not to conduct an experiment that used deception. But — in this case at least — I can’t side with the hardliners. Gwyneth’s study is really interesting. We learn a lot from it. At the same time, she demonstrates a thoughtfulness about the process that isn’t exactly typical of researchers of African Politics (some of that you see in her piece in TEPS, some of that I got from talking to her about the project during breaks at WGAPE, and some of that is just because she’s a student of Evan Lieberman and he’s a really stand-up guy). So, in the end, I hope someone like Rick Wilson or Eric Dickson reads Gwyneth’s manuscript wherever it is under review — and fairly judges the quality of the paper as well as whether deception was warranted. Dismissing it out of hand just doesn’t sit right with me.

Like this:

Related

I hate being hard-line about anything. But I can imagine that the extent of outrage at instances of scientific deception will vary depending on who they are deceiving. This isn’t a good thing. Politicians are an easy target. What if a study was deceiving a more vulnerable group? Would it’s level of interest still justify the deception? Its a slippery slope. Not saying we shouldn’t walk down it now and again. But I’d be pretty darn wary. The information gathered should be a lot more than just really interesting.