Entries tagged with “Esther Duflo”.

While behavioral economics continues to open old questions in development to new scrutiny, I am still having a lot of problems with the very unreflexive approach BE takes toward its own work (see earlier takes on this here and here). Take, for example, Esther Duflo’s recent lectures discussing mistakes the poor make. To discuss the mistakes the poor make, we must first understand what the goals of the poor are. However, I simply don’t see the behavioral economists doing this. There is still a lurking, underlying presumption that in making livelihoods decisions people are trying to maximize income and or the material quality of their lives. This, however, is fundamentally incorrect. In Delivering Development and a number of related publications (for example, here, here, and here) I have laid out how, in the context of livelihoods, material considerations are always bound up in social considerations. If you only evaluate these actions as aimed at material goals, you’ve only got a part of the picture – and not the most important part, in most cases. Instead, what you are left with are a bunch of decisions and outcomes that appear illogical, that can be cast as mistakes. Only most of the time, they are not mistakes – they are conscious choices.

Let me offer an example from Delivering Development and some of my other work – the constraint of women’s farming by their husbands. I have really compelling qualitative evidence from two villages in Ghana’s Central Region that demonstrates that men are constraining their wives’ farm production to the detriment of the overall household income. The chart below shows a plot of the size of a given farm versus its market orientation for the households operating under what I call a “diversified” strategy – where the husband farms for market sale, and the wife for subsistence (a pretty common model in sub-Saharan Africa). As you move up the Y axis, the farm gets more oriented toward market sale (1 on that scale is “eat everything”, 3 is sell and eat equally, and 5 is sell everything). Unsurprisingly, since men’s role requires them to produce for market, the size of their farm has little impact on their orientation. But look at the women’s farms – just a tenth of a hectare produces a marked shift in orientation from subsistence to market production…because women own that surplus beyond subsistence, and sell it. They take the proceeds of these sales, buy small goods, and engage in petty trading, eventually multiplying that small surplus into significant gains in income, nearly equaling their husbands. What is not to like?

Well, from the perspective of those in these villages, here is something: among the Akan, being a “good man” means being in control of the household and out-earning your wife. If you don’t, your fitness as a man gets called into question, which can cost you access to land. For wives, this is bad because they get their land through their husbands. So as a result, being in a household where the woman out-earns her husband is not a viable livelihoods outcome (as far as members of these households are concerned). Even if a man wanted to let his wife earn more money, he would do so at peril of his access to land. So he is not going to do that. What he is going to do is shrink his wife’s farm the next season to ensure she does not out-earn him (and I have three years of data where this is exactly what happens to wives who earn too much). There is a “mistake” here – some of these men underestimated their wives’ production, which is pretty easy to do under rain-fed agriculture in a changing climate. That they are this accurate with regard to land allocation is rather remarkable, really. But the decision to constrain women’s production is not a mistake, per se: it is a choice.

We can agree or disagree with the premises of these choices, and their outcomes, but labeling them as mistakes creates a false sense of simplicity in addressing problematic outcomes – because people only require “correction” to get to the outcomes we all want and need. This, in turn, rests on/reproduces a sense of superiority on the part of the researcher – because s/he knows what is best (see a previous post on this point here). That attitude, applied to the case above, would not result in a productive project design aimed at addressing income or other challenges in these villages.

Yes, people do things against material interest…but there is always a logic behind a decision, and that logic is often deeply entrenched. We would be better off talking about decisions poor people make (for better or worse), and dedicating our time to understanding why they make these decisions before we start deciding who is mistaken, and what to do about it.

I’ve just burned 15,000 words in Third World Quarterly laying out my argument for how to think about livelihoods as more than material outcomes – and how to make that vision implementable, at least via fieldwork that runs in length from days to months. I am happy to send a copy of the preprint to anyone who is interested –and I will post a version to my website shortly.

Well, the response to part one was great – really good comments, and a few great response posts. I appreciate the efforts of some of my economist colleagues/friends to clarify the terminology and purpose behind RCTs. All of this has been very productive for me – and hopefully for others engaged in this conversation.

First, a caveat: On the blog I tend to write quickly and with minimal editing – so I get a bit fast and loose at times – well, faster and looser than I intend. So, to this end, I did not mean to suggest that nobody was doing rigorous work in development research – in fact, the rest of my post clearly set out to refute that idea, at least in the qualitative sphere. But I see how Marc Bellemare might have read me that way. What I should have said was that there has always been work, both in research and implementation, where rigorous data collection and analysis were lacking. In fact, there is quite a lot of this work. I think we can all agree this is true . . . and I should have been clearer.

I have also learned that what qualitative social scientists/social theorists mean by theory, and what economists mean by theory, seems to be two different things. Lee defined theory as “formal mathematical modeling” in a comment on part 1 of this series of posts, which is emphatically not what a social theorist might mean. When I say theory, I am talking about a conjectural framing of a social totality such that complex causality can at least be contained, if not fully explained. This framing should have reference to some sort of empirical evidence, and therefore should be testable and refinable over time – perhaps through various sorts of ethnographic work, perhaps through formal mathematical modeling of the propositions at hand (I do a bit of both, actually). In other words, what I mean by theory (and what I focus on in my work) is the establishment of a causal architecture for observed social outcomes. I am all about the “why it worked” part of research, and far less about the “if it worked” questions – perhaps mostly because I have researched unintended “development interventions” (i.e. unplanned road construction, the establishment of a forest reserve that alters livelihoods resource access, etc.) that did not have a clear goal, a clear “it worked!” moment to identify. All I have been looking at are outcomes of particular events, and trying to establish the causes of those outcomes. Obviously, this can be translated to an RCT environment because we could control for the intervention and expected outcome, and then use my approaches to get at the “why did it work/not work” issues.

It has been very interesting to see the economists weigh in on what RCTs really do – they establish, as Marc puts it, “whether something works, not in how it works.” (See also Grant’s great comment on the first post). I don’t think that I would get a lot of argument from people if I noted that without causal mechanisms, we can’t be sure why “what worked” actually worked, and whether the causes of “what worked” are in any way generalizable or transportable. We might have some idea, but I would have low confidence in any research that ended at this point. This, of course, is why Marc, Lee, Ruth, Grant and any number of other folks see a need for collaboration between quant and qual – so that we can get the right people, with the right tools, looking at different aspects of a development intervention to rigorously establish the existence of an impact, and the establish an equally rigorous understanding of the causal processes by which that impact came to pass. Nothing terribly new here, I think. Except, of course, for my continued claim that the qualitative work I do see associated with RCT work is mostly awful, tending toward bad journalism (see my discussion of bad journalism and bad qualitative work in the first post).

But this discussion misses a much larger point about epistemology – what I intended to write in this second part of the series all along. I do not see the dichotomy between measuring “if something works” and establishing “why something worked” as analytically valid. Simply put, without some (at least hypothetical) framing of causality, we cannot rigorously frame research questions around either question. How can you know if something worked, if you are not sure how it was supposed to work in the first place? Qualitative research provides the interpretive framework for the data collected via RCT4D efforts – a necessary framework if we want RCT4D work to be rigorous. By separating qualitative work from the quant oriented RCT work, we are assuming that somehow we can pull data collection apart from the framing of the research question. We cannot – nobody is completely inductive, which means we all work from some sort of framing of causality. The danger is when we don’t acknowledge this simple point – under most RCT4D work, those framings are implicit and completely uninterrogated by the practitioners. Even where they come to the fore (Duflo’s 3 I s), they are not interrogated – they are assumed as framings for the rest of the analysis.

If we don’t have causal mechanisms, we cannot rigorously frame research questions to see if something is working – we are, as Marc says, “like the drunk looking for his car keys under the street lamp when he knows he lost them elsewhere, because the only place he can actually see is under the street lamp.” Only I would argue we are the drunk looking for his keys under a streetlamp, but he has no idea if they are there or not.

In short, I’m not beating up on RCT4D, nor am I advocating for more conversation – no, I am arguing that we need integration, teams with quant and qual skills that frame the research questions together, that develop tests together, that interpret the data together. This is the only way we will come to really understand the impact of our interventions, and how to more productively frame future efforts. Of course, I can say this because I already work in a mixed-methods world where my projects integrate the skills of GIScientists, land use modelers, climate modelers, biogeographers and qualitative social scientists – in short, I have a degree of comfort with this sort of collaboration. So, who wants to start putting together some seriously collaborative, integrated evaluations?

Those following this blog (or my twitter feed) know that I have some issues with RCT4D work. I’m actually working on a serious treatment of the issues I see in this work (i.e. journal article), but I am not above crowdsourcing some of my ideas to see how people respond. Also, as many of my readers know, I have a propensity for really long posts. I’m going to try to avoid that here by breaking this topic into two parts. So, this is part 1 of 2.

To me, RCT4D work is interesting because of its emphasis on rigorous data collection – certainly, this has long been a problem in development research, and I have no doubt that the data they are gathering is valid. However, part of the reason I feel confident in this data is because, as I raised in an earlier post, it is replicating findings from the qualitative literature . . . findings that are, in many cases, long-established with rigorously-gathered, verifiable data. More on that in part 2 of this series.

One of the things that worries me about the RCT4D movement is the (at least implicit, often overt) suggestion that other forms of development data collection lack rigor and validity. However, in the qualitative realm we spend a lot of time thinking about rigor and validity, and how we might achieve both – and there are tools we use to this end, ranging from discursive analysis to cross-checking interviews with focus groups and other forms of data. Certainly, these are different means of establishing rigor and validity, but they are still there.

Without rigor and validity, qualitative research falls into bad journalism. As I see it, good journalism captures a story or an important issue, and illustrates that issue through examples. These examples are not meant to rigorously explain the issue at hand, but to clarify it or ground it for the reader. When journalists attempt to move to explanation via these same few examples (as far too often columnists like Kristof and Friedman do), they start making unsubstantiated claims that generally fall apart under scrutiny. People mistake this sort of work for qualitative social science all the time, but it is not. Certainly there is some really bad social science out there that slips from illustration to explanation in just the manner I have described, but this is hardly the majority of the work found in the literature. Instead, rigorous qualitative social science recognizes the need to gather valid data, and therefore requires conducting dozens, if not hundreds, of interviews to establish understandings of the events and processes at hand.

This understanding of qualitative research stands in stark contrast to what is in evidence in the RCT4D movement. For all of the effort devoted to data collection under these efforts, there is stunningly little time and energy devoted to explanation of the patterns seen in the data. In short, RCT4D often reverts to bad journalism when it comes time for explanation. Patterns gleaned from meticulously gathered data are explained in an offhand manner. For example, in her (otherwise quite well-done) presentation to USAID yesterday, Esther Duflo suggested that some problematic development outcomes could be explained by a combination of “the three I s”: ideology, ignorance and inertia. This is a boggling oversimplification of why people do what they do – ideology is basically nondiagnostic (you need to define and interrogate it before you can do anything about it), and ignorance and inertia are (probably unintentionally) deeply patronizing assumptions about people living in the Global South that have been disproven time and again (my own work in Ghana has demonstrated that people operate with really fine-grained information about incomes and gender roles, and know exactly what they are doing when they act in a manner that limits their household incomes – see here, here and here). Development has claimed to be overcoming ignorance and inertia since . . . well, since we called it colonialism. Sorry, but that’s the truth.

Worse, this offhand approach to explanation is often “validated” through reference to a single qualitative case that may or may not be representative of the situation at hand – this is horribly ironic for an approach that is trying to move development research past the anecdotal. This is not merely external observation – I have heard from people working inside J-PAL projects that the overall program puts little effort into serious qualitative work, and has little understanding of what rigor and validity might mean in the context of qualitative methods or explanation. In short, the bulk of explanation for these interesting patterns of behavior that emerges from these studies resorts to uninterrogated assumptions about human behavior that do not hold up to empirical reality. What RCT4D has identified are patterns, not explanations – explanation requires a contextual understanding of the social.

Coming soon: Part 2 – Qualitative research and the interpretation of empirical data

You know, qualitative social scientists of various stripes have long complained of their marginalization in development. Examples abound of anthropologists, geographers, and sociologists complaining about the influence of the quantitatively-driven economists (and to a lesser extent, some political scientists) over development theory and policy. While I am not much for whining, these complaints are often on the mark – quantitative data (of the sort employed by economists, and currently all the rage in political science) tends to carry the day over qualitative data, and the nuanced lessons of ethnographic research are dismissed as unimplementable, ideosyncratic/place-specific, without general value, etc. This is not to say that I have an issue with quantitative data – I believe we should employ the right tool for the job at hand. Sadly, most people only have either qualitative or quantitative skills, making the selection of appropriate tools pretty difficult . . .

But what is interesting, of late, is what appears to be a turn toward the lessons of the qualitative social sciences in development . . . only without actually referencing or reading those qualitative literatures. Indeed, the former quantitative masters of the development universe are now starting to figure out and explore . . . the very things that the qualitative community has known for decades. What is really frustrating and galling is that these “new” studies are being lauded as groundbreaking and getting great play in the development world, despite the fact they are reinventing the qualitative wheel, and without much of the nuance of the current qualitative literature and its several decades of nuance.

What brings me to today’s post is the new piece on hunger in Foreign Policy by Abhijit Banerjee and Esther Duflo. On one hand, this is great news – good to see development rising to the fore in an outlet like Foreign Policy. I also largely agree with their conclusions – that the poverty trap/governance debate in development is oversimplified, that food security outcomes are not explicable through a single theory, etc. On the other hand, from the perspective of a qualitative researcher looking at development, there is nothing new in this article. Indeed, the implicit premise of the article is galling: When they argue that to address poverty, “In practical terms, that meant we’d have to start understanding how the poor really live their lives,” the implication is that nobody has been doing this. But what of the tens of thousands of anthropologists, geographers and sociologists (as well as representatives of other cool, hybridized fields like new cultural historians and ethnoarchaeologists). Hell, what of the Peace Corps?

Whether intentional or not, this article wipes the qualitative research slate clean, allowing the authors to present their work in a methodological and intellectual vacuum. This is the first of my problems with this article – not so much with its findings, but with its appearance of method. While I am sure that there is more to their research than presented in the article, the way their piece is structured, the case studies look like evidence/data for a new framing of food security. They are not – they are illustrations of the larger conceptual points that Banerjee and Duflo are making. I am sure that Banerjee and Duflo know this, but the reader does not – instead, most readers will think this represents some sort of qualitative research, or a mixed method approach that takes “hard numbers” and mixes it in with the loose suppositions that Banerjee and Duflo offer by way of explanation for the “surprising” outcomes they present. But loose supposition is not qualitative research – at best, it is journalism. Bad journalism. My work, and the work of many, many colleagues, is based on rigorous methods of observation and analysis that produce validatable data on social phenomena. The work that led to Delivering Development and many of my refereed publications took nearly two years of on-the-ground observation and interviewing, including follow-ups, focus groups and even the use of archaeology and remotely-sensed data on land use to cross-check and validate both my data and my analyses.

The result of all that work was a deep humility in the face of the challenges that those living in places like Coastal Ghana or Southern Malawi manage on a day-to-day basis . . . and deep humility when addressing the idea of explanation. This is an experience I share with countless colleagues who have spent a lot of time on the ground in communities, ministries and aid organizations, a coming to grips with the fact that massively generalizable solutions simply don’t exist in the way we want them to, and that singular interventions will never address the challenges facing those living in the Global South.

So, I find it frustrating when Banerjee and Duflo present this observation as in any way unique:

What we’ve found is that the story of hunger, and of poverty more broadly, is far more complex than any one statistic or grand theory; it is a world where those without enough to eat may save up to buy a TV instead, where more money doesn’t necessarily translate into more food, and where making rice cheaper can sometimes even lead people to buy less rice.

For anyone working in food security – that is, anyone who has been reading the literature coming out of anthropology, geography, sociology, and even some areas of ag econ, this is not a revelation – this is standard knowledge. A few years ago I spent a lot of time and ink on an article in Food Policy that tried to loosely frame a schematic of local decision-making that leads to food security outcomes – an effort to systematize an approach to the highly complex sets of processes and decisions that produce hunger in particular places because there is really no way to get a single, generalized statistic or finding that will explain hunger outcomes everywhere.

In other words: We know. So what do you have to tell us?

The answer, unfortunately, is not very much . . . because in the end they don’t really dive into the social processes that lead to the sorts of decisions that they see as interesting or counterintuitive. This is where the heat is in development research – there are a few of us working down at this level, trying to come up with new framings of social process that move us past a reliance solely on the blunt tool of economistic rationality (which can help explain some behaviors and decisions) toward a more nuanced framing of how those rationalities are constructed by, and mobilize, much larger social processes like gender identification. The theories in which we are dealing are very complex, but they do work (at least I think my work with governmentality is working – but the reviewers at Development and Change might not agree).

And maybe, just maybe, there is an opening to get this sort of work out into the mainstream, to get it applied – we’re going to try to do this at work, pulling together resources and interests across two Bureaus and three offices to see if a reframing of livelihoods around Foucault’s idea of governmentality can, in fact, get us better resolution on livelihoods and food security outcomes than current livelihoods models (which mostly assume that decisionmaking is driven by an effort to maximize material returns on investment and effort). Perhaps I rest too much faith on the idea of evidence, but if we can implement this idea and demonstrate that it works better, perhaps we will have a lever with which to push oversimplified economistic assumptions out of the way, while still doing justice to the complexity of social process and explanation in development.

Esther Duflo responded in a comment below – you should read it. She is completely reasonable, and lays out a clearer understanding of the discipline and her place in it than did the article I am critiquing here. I have edited the post below to reflect what I think is a more fair reading of what went wrong in the article

Doing her PhD at MIT, she was one of the first doctoral students to apply economics to development, linking the two, at a time when there were few university faculties devoted to the subject.

“It was not considered a fancy area of study,” she says. “There was a generation of people who had started looking at development from other fields. They had their own theories and only a few were economists. What I contributed to doing was to start going into detail. But I did have some advisers and mentors.”

Er, no. Development economics as a formal field had been around since the early 1980s (Note: Marc Bellemare and Duflo have both pointed out that the real roots of this discipline go back to the 1940s), and economists had been working on development issues since . . . colonialism, actually. I imagine there are a lot of senior Ph.D. economists at the IMF, World Bank and various other organizations who will be amused to hear that they were beaten to their degrees by Duflo. She was not at all one of the first doctoral students to work on this, and there are/were plenty of faculties that look at development economics.

I suspect that this might have something to do with what Mark Blaug was talking about in his article “No History of Ideas, Please, We’re Economists.” In short, one of Blaug’s arguments is that disciplinary history has largely disappeared from doctoral programs in economics, with the predictable effect of dooming the discipline to repeat its errors. I would extend Blaug’s point to many who work in the larger field of development – we have a lot of technical specialists out there with excellent training and experience, but relatively few of them understand development as a discipline with a history and a philosophy. As a result, we see “new” projects proposed and programmed despite their odd resemblance to previous efforts that failed.

There is a hint of this in the article – after all, Duflo is correct in noting that she emerged as an academic at a time when other social science fields were on the ascendancy, butshe the Guardian fails to ask why this was the trend at the time – especially after economics’ dominance of the field for so long. A little disciplinary history here would have helped – these other fields rose to prominence in the aftermath of the collapse of development economics as a formal field in the late 1980s…

So, Guardian, anyone over there actually schooled in development? Or interested in fact-checking?