Conversations and debates on issues of data access and research transparency will continue at this week’s APSA convention at various panels and roundtables, some filled with JETS advocates, others with critics, others with both. These conversations may affect decisions regarding journals’ policies. We urge you to participate in these discussions.

We append a list of events below, with the list of panelists and excerpts from panel descriptions (from the APSA online program).

Sincerely,

Nancy Hirschmann

Mala Htun

Jane Mansbridge

Lisa Wedeen

Elisabeth Wood

APSA 2016 Panels and Roundtables

Data Access, Transparency, and Replication in Political Science

Thu, September 1, 8:00 to 9:30am, PCC, 114

Panelists will discuss emerging norms, standards, and ongoing concerns in how the discipline handles data access, transparency, and replication.

Theme Panel

Chair William G. Jacoby, Michigan State University

Presenters

Raymond Duch, CESS Nuffield

Nicole Janz, University of Nottingham

Arthur Lupia, University of Michigan, Ann Arbor

Neil Malhotra, Stanford University

Colin Elman, Syracuse University

The Qualitative Transparency Deliberations: An Interim Report

Thu, September 1, 10:00 to 11:30am, PCC, 204-C

Roundtable

This roundtable will provide an interim report and discussion on the unfolding Qualitative Transparency Deliberations (QTD) sponsored by the APSA’s Organized Section on Qualitative and Multi-Method Research. Since April 2016, the QTD has involved both far-reaching consultation and intensive deliberation about the benefits, costs, and practicalities of achieving transparency for diverse forms of qualitative empirical research, including process tracing, interpretive inquiry, comparative-historical research, and ethnography. On this panel, members of the QTD Steering Committee will discuss the structure and nature of the deliberative process and the key issues on which the QTD working groups have been focusing their attention, and will provide a preview of some of the approaches to transparency and recommendations emerging from the working-group discussions.

Chair Kimberly J. Morgan, George Washington University

Presenters

Tim Buthe, Duke University

Alan M. Jacobs, University of British Columbia

Kimberley S. Johnson, Barnard College

Sarah E. Parkinson, University of Minnesota

Andrew Bennett, Georgetown University

DA-RT and Qualitative Methods: The Case of Process Tracing

Thu, September 1, 4:00 to 5:30pm, PCC, 204-B

Roundtable

The purpose of this roundtable is to stimulate a focused conversation on what DA-RT might look like in a key part of the qualitative toolkit – process tracing.

It features participants all of whom have made significant contributions to the theory and practice of process tracing. Coming from different epistemological and substantive perspectives, they will address a set of questions surrounding DA-RT.

1) Are common community standards of DA-RT/process-tracing possible and desirable?

2) At a practical/operational level, what does DA-RT look like when applied to process tracing?

3) Given that process tracing in many contexts raises foundational ethical/human-subject concerns, how can these be reconciled with demands for greater data access?

4) What are the trade-offs involved – for graduate training; for theory development; for submissions to leading journals – in implementing greater levels of transparency?

5) Given that process tracing is already a time and resource intensive endeavor, how can we raise transparency standards without simultaneously creating strong disincentives for researchers (especially younger scholars) to use the method?

6) Can formalization – through Bayesian analysis, say – enhance the transparency of process tracing? If so, are there trade-offs to consider?

The roundtable’s goal is not consensus. Rather, we hope to stimulate a discussion that gives us a better net assessment of the pluses and minuses of implementing DA-RT.

Chair Jeffrey T. Checkel, Simon Fraser University

Presenters

Andrew Bennett, Georgetown University

James Mahoney, Northwestern University

Tasha A. Fairfield, London School of Economics

Regina A. Bateson, MIT

Peter A. Hall, Harvard University

Elisabeth Jean Wood, Yale University

Roundtable on the CPS Special Issue on Transparency in the Social Sciences

Fri, September 2, 2:00 to 3:30pm, Marriott, Room 406

…As the first journal issue in political science based on results-blind peer review, the CPS Special Issue on Transparency in the Social Sciences offers a unique opportunity to critically reflect on these questions. The CPS Special Issue includes three articles that were accepted on either the basis of a prospective research design (i.e. a pre-analysis plan) or on a results-blind basis, with any discussion of the results stripped out of the manuscript. Authors of articles included in the Special Issue have also committed to make available all background information, including opening up field notes and last-minute alterations to the research design.

This roundtable brings together the participants in the Special Issue, including the Special Issue editors, CPS editors, and article authors, along with other leaders in the movement for greater transparency in the social sciences, to reflect on and debate the tradeoffs of journal publication via a results-blind process.

Chair Lauren Prather, University of California, San Diego

Presenters

Aaron S. Erlich, McGill University

Daniel Hidalgo, Massachusetts Institute of Technology

Dominika Kruszewska, Harvard University

Arthur Lupia, University of Michigan, Ann Arbor

Thomas Pepinsky, Cornell University

David J. Samuels, University of Minnesota, Twin Cities

Yael Zeira, University of Mississippi

New Tools and Standards for Research Transparency

Sat, September 3, 10:00 to 11:30am, Marriott, Franklin 11

Full Panel

The transparency revolution underway across many of the social sciences promises to improve qualitative, quantitative, and experimental research. This push is motivated in part by concerns around high levels of researcher discretion and perverse career incentives to publish striking and novel results. Deliberate or inadvertent adjustment of analysis strategies ex-post can give rise to spurious results. Despite intense interest in research transparency, the gains from this movement are not yet fully realized, due in part to the lack of shared standards and practices. This panel comprises four concrete proposals for improving ex-ante and ex-post transparency that cut across research traditions and empirical subfields.

Lin and Green address the problem that even detailed research plans inevitably fail to address unforeseen complications. Results are often heavily dependent on the choices researchers make in these circumstances. Lin and Green propose that researchers develop standard operating procedures in order to reduce researcher discretion.

Elman, Kapiszewski, and Lupia provide criteria that researchers can use to demonstrate that they followed the rules of their respective logics of inquiry. Importantly, they clarify the distinction between transparency at the production stage and at the analytic stage.

Transparency standards are particularly underdeveloped for computer-assisted text analysis methods, a relatively new area of inquiry. Romney, Stewart, and Tingley outline a new set of tools that will increase the confidence in and reproducibility of text analysis from data collection to statistical analysis.

Blair, Cooper, Coppock, and Humphreys build a framework for completely characterizing the formal features of a research design: the population, sampling strategy, potential outcomes, assignment mechanism, estimand, and estimation strategy. The tools, including software, they propose can be used to specify, improve, and pre-register designs.

Who Publishes, Who Perishes? Challenges in Research with Vulnerable Populations

Sat, September 3, 4:00 to 5:30pm, Loews, Commonwealth C

Roundtable

Scholars across the various fields of political science and those involved in interdisciplinary research with colleagues in anthropology, history, sociology, social work, and related disciplines here and abroad face a range of ethical constraints when studying highly vulnerable individuals and groups.

…

Scholars who conduct research with highly vulnerable populations are bound to safeguard the integrity of their data in order to ensure the wellbeing of people who could be targeted for reprisals if their identity were revealed. Yet these same scholars are faced with the prospect of making interview and focus group transcripts, field notes and other research data accessible and/or more public and transparent in order to fulfill the DA-RT requirements and to disseminate their work through the publication of scholarly articles (a particular concern for junior scholars).

This roundtable brings together scholars to discuss the ethical implications of this new research and policy landscape. We will pay particular attention to the potential effects these twin pressures may have on scholars involved in interdisciplinary work and the gendered and/or racialized implications of this controversy for scholars themselves as well as the people whose reality we study at the grassroots level.

Chair: Shareen Hertel (University of Connecticut)

Presenters:

Elisabeth Wood (Yale University)

Lisa Wedeen (U-Chicago)

Jane Mansbridge (Harvard JFK School)

Sarah Parkinson (U-Minnesota Humphrey School)

Deborah Yashar (Princeton University)

Structures of Political Science Revolutions: DA-RT and Audit Cultures

Sunday, September 4, 8-9.30 a.m., Loews, Commonwealth A2

Roundtable participants’ shared concern for the future of political scientific practices leads them to consider the following sorts of questions. What is DA-RT’s transformational potential vis à vis institutional forms and strategic relationships? What is the political science future that its proponents envision? Instead of uniting the discipline behind (or under) a single procedural banner, does DA-RT have the potential to split the discipline into, say, “DA-RT journals” versus “traditional journals”; or to split the Association itself because of concerns over how it has been introduced (neither of these notions so farfetched in light of the UK’s experience; see Shore and Wright 2000)? Is the current framing of DA-RT a part of the neoliberal reforms ongoing in higher education worldwide (journal impact scores, measures of faculty “productivity” [e.g., the US’s Academic Analytics, the UK’s Research Excellence Framework]), replacing discussions of quality with depoliticized systems to monitor it (Pels 2000:135)? And what about its relationship to philosophy of science debates concerning the unity of science or to the actual status of replication in natural science, a status presumed by policy proponents?

Roundtable participants will speak to these and other questions. In keeping with the minutes from both 2012 and subsequent Executive Council meetings calling for discussion of DA-RT with the membership and calls across the discipline to have the conference meetings this year enable further, explicit discussion of the framing and desirability of such research and editorial policies, the roundtable is intended to develop a conversation that opens up new issues for exploration.

Starting immediately, AJPS will award “open data” and “open materials” badges to articles that make publicly available “the digitally-shareable data necessary to reproduce the reported results” and “the components of the research methodology needed to reproduce the reported procedure and analysis,” respectively. These are two of the “Badges to Acknowledge Open Practices” from the Center for Open Science (COS). See announcement here.

In a post on his blog, Tom Pepinsky raises questions about DA-RT and expresses concerns that the QTD process is reifying the dichotomy between “quantitative” and “qualitative” research. Andy Moravcsik writes a long response affirming that active citation right now amounts to the “only viable default approach for qualitative work.” Active citation, he claims, is a very conservative “back to the future” proposal” that would make journal articles in qualitative political science resemble articles of three decades ago, with “discursive footnotes, textual references, longer word-limits, and interpretive analysis.”

The Association for the Study of Nationalities (ASN) has recently learned of the DA-RT discussions originating within the American Political Science Association (APSA). We are concerned that a set of stipulations regarding research signed by 27 journals (JETS) was enacted on January 15th 2016 when so few of those who will be affected by these stipulations are aware of their requirements. We share the concerns outlined by the 1,173 petition signatories to delay DA-RT implementation to more fully consider its implications for a broad range of research.

The DA-RT stipulations establish conditions in which editors and reviewers can request that materials produced in the course of research should be made available on a public domain website. These materials can include databases but also interview transcripts, recordings, and fieldnotes. We understand that exemptions can be requested from these requirements. However, we remain concerned that there is not sufficient clarity in the policies as outlined to protect from the potential harms to sources and researchers in the conduct of research, as well as to long-term implications of harm for the conduct of research on sensitive topics. Nor do we find IRB / ethics board statements to be sufficient protection, given the diversity of access to such protections for our members across the globe. We note many of the concerns DA-RT raises for research on violence and autocratic regimes as well as those using qualitative approaches, as outlined in the APSA Qualitative and Multi-Methods Research newsletter in 2015 as well as the APSA Comparative Politics Newsletter in Spring 2016. As researchers on sensitive topics, we share many of these concerns in our own work. We are particularly disturbed that the serious problems outlined by researchers in these forums have not yet produced changes to the policy appropriate to the potential harms.

We urgently request a re-thinking of the DA-RT stipulations to incorporate better policies to address concerns of researchers working on sensitive topics, including those using qualitative and historical research.

The conference on “Conducting Fieldwork under Complicated Circumstance,” organized by two graduate students at the University of New Mexico (Anna Calasanti and Fiorella Vera-Adrianzén) will involve faculty, students, and other practitioners in two days of dialogue about strategies and techniques to manage ethical, logistical, and personal security issues during fieldwork. Check out the conference website here. Topics to be discussed include:

Ethical

Risk of revictimization/ retraumatization of research participants

Instrumentalization of research participants

Externalities (i.e. risks and benefits) of participation in fieldwork activities

Disruption or transgression/violation of cultural norms

Pros and cons of cultural embedding, building trust and personal relationships with participants

Giving back to the community: why and how should we give back to the community? What are some of the constraints, benefits and risks when doing so? What are some appropriate ways to do it?

Mediator role among parties that are in conflict in the field: Shuttle diplomacy

Intervening to help research participants (directly or indirectly)

Confidentiality and privacy of research participants’ responses: how to secure this at all times, especially when methods imply a bigger audience (i.e. focus groups)? Are there any cases in which these principles can be overlooked? What should be done under those circumstances?

DA-RT (Data Access and Research Transparency): pros and cons

Identity of the researcher

Perceptions about researcher: What kind of impression do we give social actors about us, as researchers, as representatives of academic institutions or of disciplinary fields?

An open deliberation to develop community standards of research transparency (QTD process) begins today. Please register and participate here. You do not need to belong to APSA’s QMMR section in order to participate.