Public Policy News

The Internal and External Validity of the Regression Discontinuity Design: A Meta-Analysis of 15 Within-Study-Comparisons | JPAM Featured Article

Regression discontinuity (RD) is generally acknowledged as the most rigorous non-experimental method for obtaining internally valid impact estimates. The study tests the efficacy of RD by comparing RD causal estimates at the treatment cutoff to those from Randomized Control Trials also estimated at this same cutoff. The study identifies 15 previously completed within-study-comparisons that explicitly examined this issue by assuming the RCT results are unbiased and comparing them to RD results.

Coverage of the APPAM/MDRC Institutional Forum

First Panel: "Family & Work"

In a high-energy session chaired by Emily Schmitt, Administration for Children and Families, Nadine Dechusay, MDRC’s Center for Applied Behavioral Science (CABS) and Ariel Kalil, University of Chicago’s Behavioral Insights and Parenting Lab, discussed the application of behavioral science to influence of parenting gaps, choices, and habits. Lisa Gennetian, New York University & beELL Initiative, wrapped up the discussion with general takeaways and how the energy of the behavioral science could be pushed through to families with vulnerable children.

At CABS, Dechausay focuses on testing applications of behavioral insights to simplify and improve insights in government programs. Specifically, she analyzed the process of recertification for low income families seeking subsidized child care. The process includes application, enrollment, finding a provider, and recertification, which seems simple but many challenges exist in reality. These families are not always made aware of these subsidy programs, often face waitlists for participation, are provided a very complex requirements form, and must attend multiple in-person meetings for recertification. Given these obstacles, nearly 17% of participants do not renew their eligibility. CABS performed qualitative and quantitative studies in Indiana and Oklahoma in order to identify the gaps in the recertification process and how they can be narrowed. In both studies, participants were given highly-simplified requirement forms, recertification appointment reminder forms, and shorter lists of proper documentation needs. The studies found that by redesigning the implementation of these processes, the number of participants that renewed on time and with only one appointment increased significantly. The results showed that by changing the method of communication, redesigning forms, and altering the contact timelines, the studies were ultimately successful because they changed the parental habits.

Kalil, a developmental psychologist, then discussed what she believes is the key to closing the gap in child outcomes caused by how parents engage with their children. Society typically believes that low parent-child engagement is caused by go-to reasons such as poor access to books, lack of time, and lack of understanding in child development, but Kalil offered an alternative view that parenting decisions are often shaped by cognitive biases. She believes that parents make choices today that aren’t necessarily consistent with how they see themselves and their children in the future. Parents often tend to make heuristic decisions that are fast, cheap, and don’t require deeper thinking. Using this approach, Kalil discussed The Parents and Children Together (PACT) intervention which sought to overcome “present bias” and bring future aspirations to the present. The intervention provided families with iPads, loaded with just a single app for reading, for six weeks and tracked how many minutes the parents read with their children. One group of families was simply told to use the app as they pleased, the other group was provided with nudges, goals, feedback, and rewards for their participation. The result was that the incentivized group read 2.5 times more minutes than those that were not. Khalil found that by encouraging parents to change their engagement habits, they were able to align ‘aspirational parenting’ with ‘actual parenting’.

Gennetian concluded the session by reiterating the importance of using behavioral science to better understand actual families and restricting their environment and habits in order to help them create better parenting habits. These studies must continue to provide low-cost, quick, and real-time results if they wish to remain successful, but it’s also important to ensure that researchers are converting behavioral assumptions into the right questions.

Keynote Speech

Nathaniel Higgins, a fellow on the Social and Behavioral Science Team (SBST), provided a short keynote address. SBST is a subcommittee of the National Science and Technology Council and represents more than 25 organizations across government including the Departments of Defense, Agriculture, Veterans Affairs, Health and Human Services, and Education. A diverse group of civil servants, policymakers, and experts in a variety of fields have collaborated to complete more than 30 pilots with rigorous evaluations.

Higgins talked about the creation of the SBST and what they are working on now. The creation of the team was based on a four-part executive order and the four parts are the basis of the focus of the SBST. The team is currently looking at opt in versus opt out preferences on government forms – does changing the format of the question translate to increased enrollment or utilization of certain programs?

The lens through which the SBST views their work is based on improving access to programs, promoting good choices and providing incentives. They hope that through these three key variables, they can enact substantive and positive change.

Higgins also talked about opportunities for the future in this arena, particularly, cross agency efforts, like the effort to share data and resources between the Census Bureau and the White House. He also talked about agency leadership making progress, especially USAID and HUD. State and city partnerships are also burgeoning, including the Philadelphia Behavioral Science Initiative, the Lab at DC, Urban Labs in Chicago, ideas42 and others.

Second Panel: "Youth"

The second panel of the day focused on applying behavioral science to low-income and vulnerable youth. The session was chaired by Caitlin Anzelone from MDRC’s Center for Applied Behavioral Science and the panelists included Ben Castleman from the University of Virginia and David Yeager from the University of Texas at Austin and the Mindset Scholars Network. The discussant for the session was Brigitte Madrian from Harvard University.

Castleman focused on “nudges” in education, specifically text messages that were impactful in helping vulnerable parents. He cited the “Ready for K!” text message as a promising experiment that helps parents prepare kids to read. There is promising evidence from text nudges RCT’s and these experiments will be examined more in the future. Largely because they are a low cost, high impact, scalable interventions that allow for creative and clever designs. What needs to be addressed now is the lasting positive impacts, if any, and whether the results are replicable? Researchers need to be aware that the medium is always changing and interventions on Kik, Snapchat and What’s App might be on the horizon soon.

Yeager discussed the learning of mindsets and how mindsets affect choice and behavior. Phase I of his work was replicating mindset interventions and Phase II was replicating intentionally to learn about treatment effect heterogeneity. The challenges for researchers are getting the intervention right for the right population. A lot of progress has been made but more work must be done.

Madrian, acting as a discussant, focused on an overview of the field. She said testing an evaluation is important, people don’t always behave in ways you would predict. In testing an evaluation, they often find that different barriers exist for different people. She discussed what might happen with SBST office after the new administration comes to DC. While Madrian was not sure what might happen, she did stress that research like this should not be political. Further, Madrian discussed the perception of behavioral science as a tool for liberals. She discussed that a tool cannot be ideological in nature and while it could be used for liberal pursuits, it could be used for conservative pursuits as well. Behavioral insights can be unethical, researchers need to be careful with structure (presumed consent, opt out or opt in, for example). Finally, she mentioned that she though the term “nudges” sounded both manipulative and small. And the effects of a “nudge” are not small.