Public Policy News

The Internal and External Validity of the Regression Discontinuity Design: A Meta-Analysis of 15 Within-Study-Comparisons | JPAM Featured Article

Regression discontinuity (RD) is generally acknowledged as the most rigorous non-experimental method for obtaining internally valid impact estimates. The study tests the efficacy of RD by comparing RD causal estimates at the treatment cutoff to those from Randomized Control Trials also estimated at this same cutoff. The study identifies 15 previously completed within-study-comparisons that explicitly examined this issue by assuming the RCT results are unbiased and comparing them to RD results.

Going Local

September 4, 2014 09:00 AM

By Scott W. Allard, University of Washington Evans School of Public Affairs

As anyone who has done a local site visit or studied street-level bureaucracy knows, there is a lot of interesting variation in the delivery of policy or programs at the local level. Local idiosyncrasies of political conflict or will, economic conditions, and bureaucratic or nonprofit capacity matter to the outcomes we observe across places, programs, and populations. To some extent, “all policy is local.”

Although well-designed case studies, ethnographies, or surveys can yield the best insight into the localness of policy and programs, even brief explorations of local context around a given policy topic or phenomenon – “going local,” if you will – can be a really useful exercises for someone in the early stages of project development.

Not to be confused with careful qualitative study design, thoughtful in-depth interviews, or well-tuned survey instruments, I find going local to be more like informal policy research reconnaissance. It’s an approach that many of us find useful from time to time, as it can help us think about key conceptual or methodological issues relevant to the more elaborate rigorous study designs we pursue.

Going local can be many things. It can be driving around a little and talking to a few key informants or attending some community meetings. It may be a chance visit to a local library or service provider. Even eating a meal at a family-owned local joint and checking your priors while standing in line talking to local residents.

In particular, I have found “going local” useful for generating insight into the local-level economic, demographic, political, and organizational realities that have profound importance on policy or program delivery across larger geographic areas. Insights that while relevant to national level policy decisions, would not be immediately apparent through national-level data sources. Insights that help lay the groundwork for more scientific research designs and analyses.

For example, early in my career, I was working on a project examining place, poverty, and human service provision in the City of Detroit. The project was applying the labor market spatial mismatch theory to human service program participation - we expected to find spatial proximity to human service organizations related to human service program take up. While working on a JPAM revise and resubmit testing these ideas, I had the chance to engage a few local nonprofit human service organizations around graduate student capstone consulting projects. I remember being struck at how the changing nature of place was affecting the work of these organizations. One organization had to reinvent itself as the community around it gentrified. Its service mission shifted from providing mental health services to low-income adults to specializing in treating eating disorders for more affluent adolescent youth. Another student team worked with an organization to revise an outdated fundraising strategy. Middle-class population flight to the suburbs reduced the flow of private donations to support program activities, requiring the organization to find new and more reliable revenue sources. Throughout each instance, these local incursions helped me think about the relationship between place and the provision of social services. These relationships, while observed in a specific local place, seemed to occur similarly in a variety of different places. Thus, I began to develop the outline for a large-scale survey of service providers that would be central to my first book on the geography of the human service sector.

Later, as I was finishing that book, I took a visit to a suburban food pantry on a whim. The pantry manager’s commented that her pantry’s caseload was on pace to double from the previous year, despite her location in a seemingly affluent suburb and at a time when the national economy was in a state of slow growth. I quickly followed up with a few other suburban human service providers and found many of the same responses. What these suburban nonprofits were picking up was the beginning of the Great Recession and the tipping point where poverty had become more suburbanized than ever before in American history. Again, “going local” led to insights that launched new projects using new primary and existing secondary data to focus on the changing metropolitan geography of poverty in the US and urban-suburban differences in access to food resources.

Now that these projects are finishing up and hopefully will appear in journals and bookshelves near you someday soon, I've started to think about my next trip - how will I “go local” and what will I learn?

*****

Scott W. Allard is a Professor of Public Affairs at the University of Washington Evans School of Public Affairs with expertise in urban poverty, employment among low-skill workers, food security, safety net utilization, and the spatial accessibility of governmental and nongovernmental safety net programs. Allard has received research grants supporting his work on social welfare policy from the Department of Health and Human Services (HHS), Department of Housing and Urban Development (HUD), The Brookings Institution, the University of Kentucky Center for Poverty Research (UKCPR), and the Rural Policy Research Institute (RUPRI). He is also the author of Out of Reach: Place, Poverty, and the New American Welfare State. Allard has authored several articles on contemporary social welfare policy and social service delivery in several academic journals.