Blogs

This is part 2 of a two part blog on what survey implementers would tell the researchers and others who work with them (part one is here). Before we dive in, I want to reiterate my thank to the folks at EDI and IPA, as well as James Mewera of the Invest in Knowledge Initiative, Ben Watkins at Kimetrica, and Firman Witoelar at SurveyMeter who took the time to send me really careful thoughts and then answer my queries. As before, don’t take anything below as something specific any one of them said – I’ve edited, adjusted and merged. Blame me if you don’t like it. One final note, as you can see from the list, not everyone one of these is a commercial firm, and some of them do research as well – so not only keep that in mind when filtering the advice, but I’ll abbreviate with SO for survey organization.

Please read this post as me channeling and interpreting their voices. I am not sure I agree with everything I heard, but I am passing it on. And all of it gave me food for thought. Stuff in [italics] is me explicitly responding to a couple of points.

It’s hard to argue against the idea that giving cash to someone in need is the best you can do for that person in most circumstances: money maximizes your choice set and any conditions, strings attached, etc. makes that set smaller. With the advance of mobile technologies and better, bigger data, you can now send someone anywhere in the world money and make that person’s life instantly better – at least in the short run. But, what if I told you that with every dollar you send to one poor person, you’re taking away food from a few other people? How should we evaluate the impact of your transfer then?

Women tend to have preferences that are more pro-social and are less risk-taking and less patient on average (the latter I was surprised by, but the difference is not so large). In Science this week, Falk and Hermle look at how these gender differences in preferences are correlated with economic development and gender equality using survey data on 80,000 individuals in 79 countries! They find gender differences increase with GDP and with gender equality – their explanation is “As suggested by the resource hypothesis, greater availability of material resources removes the human need of subsistence, and hence provides the scope for attending to gender-specific preferences. A more egalitarian distribution of material and social resources enables women and men to independently express gender-specific preferences.”

I have often wondered what the folks who do the surveys I use in my research think of how it is to work with me. Since I wasn’t sure I had the courage to hear that straight to my face, I wrote to a number of survey folks I knew (and thought highly of) or that other people recommended. I asked them what they would tell researchers in general.

Governments have at least four reasons to try and bring firms into the formal system:

To broaden and increase the tax base

To enable firms to access the formal economy and help spur firm growth through the potential benefits of being formal (such as access to financial services and government contracts)

To increase the sense of rule of law by having the default be that everyone is obeying the law

To have firms provide information about themselves to the state, which can help the government better understand the structure of the economy and to better target business programs.

The most common way of trying to achieve these aims has been through regulatory reforms that make it easier for firms to formalize. This has taken the form of “one-stop-shops” which have been implemented in at least 115 countries and which enable firms to register both as a business and as a tax entity all at once. However, a number of randomized experiments that have followed such reforms have seen very few informal firms formalize. This raises the question of whether regulatory simplification alone is not enough, and whether trying to achieve all of the above four goals with one instrument causes none of them to be attained.

Separating business and tax registration, and an experiment in Malawi
In a new working paper (replication data) (joint with Francisco Campos), we conducted an experiment with informal firms in Malawi that aimed to test whether governments can bring firms into at least part of the formal system and thereby achieve at least some of the above goals, and whether firms need additional help to realize the benefits of becoming formal.

Stephanie Schwartz asks “are research ethics a question of public opinion?” on the Political Violence at a glance blog - which discusses a new study that asked both research subjects and scholars their opinions on the acceptability of different research designs. Interesting discussion, particularly around what to do when the two differ – e.g. “A human rights advocate wants their interview with a researcher to be on the record. But the researcher worries that disclosing the subject’s name might put them in harm’s way. Does the researcher follow the participant’s understanding of “acceptable risk” and publish their name? Or do they follow their own instincts and keep the source anonymous?” (h/t This Week in Africa).

On the Future Development blog, Ariel Benyishay and co-authors discuss how they used satellite data to evaluate a USAID rural roads project in Palestine – using a diff-in-diff approach they compare nightlight in 750 meter grid cells shortly before, during, and shortly after the roads were rehabilitated to those in not-yet-improved cells. The report has some discussion of the many challenges involved, such as how to interpret an increase in nightlights, dealing with cells which have multiple roads treated, and the problem of potential spatial reallocation of economic activity.

Teachers are important. And many teachers in low- and middle-income countries would benefit from support to improve their pedagogical skills. But how to do it? Again and again, evidence suggests that short teacher trainings – usually held in a central location – don’t do much of anything to improve teacher practice. Likewise, much teacher training is overly theoretical and doesn’t translate into practical pedagogical improvements.

Providing teachers with one-on-one coaching is a popular alternative. A coach comes to the classroom, observes the teacher, and provides practical feedback. It makes sense. As Kotze and others put it (in turn paraphrasing earlier authors), “Teachers only learn to do the work by doing the work, and not by being told to do the work, or being told how to do the work, or being told that they will be rewarded or punished for outcomes associated with the work.” A recent review of U.S. evidence shows big impacts of coaching on both teacher practices and on student learning. But those big effects are concentrated in small-scale programs: Effects tend to be much smaller when implemented at scale. Outside a high-income environment, a teacher coaching pilot in South Africa compared coaching to a more traditional training at a central location. Students whose teachers received coaching learned twice as much as students who teachers received training.

Vision statement from Tage Rai, the new social and behavioral sciences editor for Science on what they are looking for: “I feel that our strength is the ability to bridge across social sciences in a way that very few outlets can and at a level that none can match. Therefore, we will be emphasizing papers that cross over major fields more strongly than ever before (e.g. psychology and anthropology, economics and political science, sociology and computer science).... The other major concern that I encounter has to do with scientific practice and reproducibility concerns within the social sciences. Science is actively engaged with these issues and continues to consider best practices going forward. ... First, only a small percentage of papers are sent out for in-depth review. For these papers only, authors will be asked to upload their data to an online repository accessible to reviewers.... I will host a twitter Q&A to answer any questions people may have about publishing with Science. For example, I am often asked about formatting for Science. As most papers are rejected, my approach is to be pretty loose about formatting, with the understanding that if a paper moves further into the process, we can revisit questions of word length and formatting at that time.”

My daughter thinks math is hard. She also thinks she is not good at it. She is, by some set of objective measures, actually quite good at it. But she keeps repeating this mantra to me when we are sitting there slogging through her homework. I tried all kind of positive reinforcement and then, one day, I sat her down, and explained that there is a widely held gender bias belief about girls doing worse in math. It didn’t really make a difference.

Following on from yesterday’s post on doing development economics at a Liberal Arts college, we have a second post today to get additional perspectives. One point I wanted to note is that I think that while the post is about liberal arts colleges, many of the same issues will arise for people teaching in universities in other countries that don’t have large PhD programs, as well as some of the same issues also face researchers at the World Bank and other development research careers outside of academia. So even if you aren’t interested in liberal arts schools, read along...