WPS6722
Policy Research Working Paper 6722
The Method of Randomization
and the Role of Reasoned Intuition
Kaushik Basu
The World Bank
Development Economics Vice Presidency
Office of the Chief Economist
December 2013
Policy Research Working Paper 6722
Abstract
The method of randomization has been a major driver in it is not able to demonstrate generalized causality. Nor
the recent rise to prominence of empirical development does it, in itself, lead to policy conclusions, as is often
economics. It has helped uncover patterns and facts that claimed by its advocates. To get to policy conclusions
had earlier escaped attention. But it has also given rise to requires combining the findings of randomized
debate and controversy. This paper evaluates the method experiments with human intuition, which, being founded
of randomization and concludes that, while the method in evolution, has innate strengths. Moreover, even non-
of randomization is the gold standard for description, and randomized empirical methods combined with reasoned
does uncover what is here called ‘circumstantial causality,’ intuition can help in crafting development policy.
This paper is a product of the Office of the Chief Economist, Development Economics Vice Presidency. It is part of a larger
effort by the World Bank to provide open access to its research and make a contribution to development policy discussions
around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author
may be contacted at kbasu@worldbank.org.
The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development
issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the
names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those
of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and
its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.
Produced by the Research Support Team
The Method of Randomization and the Role of
Reasoned Intuition
[Albert Hirschman Lecture, LACEA-LAMES, Mexico City, 31 October, 2013]
Kaushik Basu
Senior Vice President and Chief Economist, World Bank
and
Professor of Economics and C. Marks Professor, Cornell University
Contact: 1818 H Street, NW
Washington, DC 20433
Email: kbasu@worldbank.org
Key words: randomization, instrumental variables, development policy, causality, intuition
JEL Classification Number: B41, O20, Z18
Acknowledgements: The method of randomization has been a topic of interest to me for quite
a while. During this long period and, more recently, in the course of preparing for the Albert
Hirschman lecture and in writing it up as a paper, I have accumulated many debts and would
like to thank, in particular, Abhijit Banerjee, Talia Bar, Alaka Basu, Karna Basu, Tito Cordella,
Karla Hoff, Vamsee Krishna Kanchi, Luis-Felipe Lopez-Calva, Mattias Lundberg, Celestin Monga,
Augusto de la Torre, Daniel Lederman, Nora Lustig, Dilip Mookherjee, Ted O’Donoghue, John
Roemer, Vito Tanzi, and Merrell Tuck-Primdahl.
Preamble
I am greatly honored to be invited to give the Albert Hirschman lecture at
the LACEA-LAMES conference in Mexico City on October 31, 2013. What those
who invited me to deliver this lecture would not have known is that I owe a
special debt to Albert Hirschman. I met Albert for the first time in early 1985
when he came to India on a visit organized by the Ford Foundation. I was familiar
with his writings and after his lecture in Delhi got into a long discussion and some
argument too. The next day, at a social gathering, he drew me aside and asked me
if I would consider spending a year at the Institute for Advanced Study in
Princeton. I answered with indecent haste. The one year, 1985-86, that I spent in
Princeton opened up for me a lasting interest in political economy and related
social sciences. Following Hirschman’s lead, and stealing a favorite expression of
his, I learned to trespass on neighboring disciplines and have never given up on
that since. My wife and I got to know him and his wife, Sarah, well and greatly
appreciated their kindness and warmth during that wonderful year. I am grateful
to be able to pay a tribute to Albert Hirschman with this lecture and have
deliberately chosen to speak on an area of wide methodological significance to
economics, especially in the context of development policymaking. The topic is
the method of randomized trials and its role in discovering causal links and
designing policy. I chose this topic somewhat self-consciously because there are
certain things in life that are easier to do than to explain how to do. Cycling and
research fall in this category. This paper is about the “how to do” of research.
1. Introduction
Human knowledge owes a lot both to the ability of human beings to think
and to their inability to think deep enough. This is not as paradoxical as it may
sound. It implies people typically know less than they think they do. A careful
examination of how we acquire knowledge, even of the most scientific nature,
compels us to recognize that a lot of what we take to be knowledge is illusory.
2
The present paper is focused on one of the most important and rapidly-spreading
scientific methods of knowledge acquisition in economics, that of randomized
trials.
The method of randomization has long been a part of the instruments of
investigation in epidemiology, and it is often treated as the gold standard for
making causal inferences. Randomization and probabilistic inference as a method
for acquiring scientific knowledge goes back to, at least, the nineteenth century.
An interesting early example of probabilistic inference occurred in studying
telepathy and parapsychology. In an experiment in 1884, Charles Richet wanted
to see if one person’s drawing and looking at a playing card and the information
thus acquired by the person gets transmitted to others. His study showed that of
the 2,927 guesses about the card, 789 were correct whereas the expected
number of correct guesses if they were completely random would have been 732.
This allowed him to conclude that there is a small and error-ridden but,
nevertheless, positive transmission of knowledge that occurs between people’s
minds. These early parapsychology studies and debates drew in some prominent
thinkers including philosophers and economists, like Henry Sidgwick and Francis
Ysidro Edgeworth. For a fascinating account of this early history of the subject,
including some proper randomization studies, see Hacking (1988) 1.
The arrival of randomization as a systematic method of investigation in
economics and, in particular, development economics, is, however, relatively
new. But it has arrived here with such gusto that this has become a major story of
our time, giving rise to widespread use around the world by development
economists, and, in equal measure, debate and discussion about its value and
validity. 2
1
As must be obvious from this paragraph, what constitutes a proper randomization experiment is not beyond
dispute. Even in modern development economics many variants, around the broad idea of randomization, have
been tried and critically evaluated (see Bruhn and McKenzie, 2009).
2
See, for instance, Banerjee, 2005; Banerjee and Duflo, 2009, 2012; Bardhan, 2013; Barrett and Carter, 2010; Basu,
2005, 2011; Cartwright, 2010; Cartwright and Hardie, 2012; Deaton, 2009, 2010; Duflo, Glennerster and Kremer,
2008; Elbers and Gunning, 2013; Heckman and Smith, 1995; Mookherjee, 2005; Ravallion 2009; and Rodrik, 2008.
3
It will be argued here that the rise of randomization and randomized
control trials (RCTs) is of immense value to economics and to the crafting of
economic policy. It gives us insights into history that we did not have earlier. At
the same time, many of the claims widely made on behalf of RCTs are
exaggerated and invalid. It will be shown that RCTs do not give proof of any
universal causality. Provided that we treat as axiomatic that the world is causal, it
enables us to establish what will be here called ‘circumstantial causality.’ Some of
the help that, in indirect ways, it provides to shaping policy is also provided by
non-randomized methods and studies of pure correlations. What RCTs do well,
and on this it is indeed the gold standard, is to describe large populations over
multiple periods. Its true value lies in good description; and the worth of good
description is not something to be dismissed (Sen, 1980). From this to go to policy
requires the use of intuition. If we deny intuition any ground, RCTs are of no
value. But the paper argues that the use of intuition is justified and, hence, RCTs
do act as ‘aid’ to policymaking.
Why do we make the mistake of thinking that studies based on exogenous
randomizations, RCTs and even the method of carefully-selected instrumental
variables give us insights into universal causality? The next three sections try to
answer this question.
2. Randomization and Causality
To understand why we often tend to think that RCTs demonstrate universal
causal links, consider two celebrated studies that used randomization and led
researchers to some striking findings--Chattopadhyay and Duflo (2004), and
Miguel and Kremer (2004). Chattopadhyay and Duflo raise a pertinent question:
Does having an elected woman leader to head a local government make a
difference to the quality of life in the village or locality? There is, it can be shown,
a positive correlation between having women leaders and the quality of provision
of public goods, such as water. But, a priori, it would be reasonable to say that
causality can run either way. On the one hand, it is possible that women have a
4
greater sense of social responsibility and so once elected they make sure that
villagers get good water. On the other hand, it is not difficult to think of the
causality running the other way around. In a society like India’s, it is traditional for
women to be in charge of getting water for the household. It is arguable that if
fetching water takes up a lot of time in a village, women are unlikely to have the
time to run for office, and so would be unlikely to be the elected head of the local
government, the panchayat.
It is therefore possible to argue, a priori, that the correlation between the
election of a woman leader and the better provision of public goods, such as
water, reflects causality running either way. Chattopadhyay and Duflo (2004)
sorted this out through a deft use of a natural randomization exercise.3 The Indian
government had decreed through a constitutional amendment that in one-third
of the panchayats all over India, selected by lottery, the leader must be a woman.
Utilizing this random choice of women leaders, they showed that one of the a
priori hypotheses, namely, that the choice of elected women leaders was caused
by the better provision of water, was false. And, from this, they concluded that it
was the choice of a woman leader that caused the better provision of public
goods and, in particular, water.
What their study did, virtually beyond question, was to rule out the
hypothesis that causality ran from villages with superior provision of water to
women being elected leaders. However, it never proved the causality ran in the
reverse direction. That impression was created in our minds by the unwitting use
of a set of prior possibilities and picking the residual after all other possibilities
were ruled out by trials and experiments. To jump from this to the policy
conclusion that we should elect a woman to the head the panchayat to improve
village water supply would be hasty.
3
This rightly-celebrated paper has spawned other papers shedding interesting light on the status of women in
some parts of India (for instance, Beamen, Chattopadhyay, Pande and Topalova, 2012; Iyer, Mishra, Topalova,
2012).
5
Similarly, consider the seminal work of Miguel and Kremer (2004). In
developing countries there tends to be a correlation between the use of
deworming medicine by school children and superior participation in schools by
children. Why may this be so? Regular participation in school may teach the
children the importance of good hygiene and, in particular, keeping oneself free
of stomach worms and so they take deworming medicines. It can be that some
households are smarter and smartness means being aware of the importance
deworming and going to school regularly. It can also be that deworming makes
children fit and fitness enables them to attend school regularly. Miguel and
Kremer used an excellently designed RCT. They administered deworming
medicine to children in a bunch of randomly selected schools in Kenya 4 and found
that participation rates of students improved. This clearly ruled out the first two
causal hypotheses. 5
It should be clear that in neither of the two studies just discussed is
causality established. Based on these findings, we cannot assert that electing a
woman leader in a Tamil Nadu village will improve water provision there and
administering deworming medicine to all school kids in a school in Kenya in 2014
will improve school attendance. All these two RCTs do—and RCTs are ideal for
this—is to rule out certain possible causal links. They give the impression of
having pinned down causality only because we carry certain priors about causal
links and when all but one of our priors are ruled out, our mind jumps to the
conclusion that causality has been established; 6 and we feel that we are now
ready to use these findings to design actual policy interventions. 7
4
In this brief statement I am, of course, leaving out important details of the experiment.
5
These are just two papers I am using for my argument. But RCTs have given us a rich haul of findings on past
regularities in a range of domains in development economics: see, for instance, Ashraf, Karlan and Yin (2006),
Bernard, Dercon, Orkin and Taffesse (2013), Habyarimana and Jack (2012), Jensen (2012), Hoff and Pandey (2006),
Ifcher and Zarghamee (2011).
6
The reason magic surprises us has roots in a similar phenomenon of the human mind. The skillful magician instills
in us certain prior beliefs about how a rabbit may have got into a hat. Then he or she demonstrates that none of
these are possible and when the rabbit is pulled out of the hat, we are left wondering if this was a miracle.
7
The journey from (allegedly) finding causes to designing policy is fraught with dangers and is the subject matter
of considerable writing (see, for instance, Cartwright, 2007; Basu, 2000; Das, Devarajan and Hammer, 2011). I shall
return to this later.
6
To be aware of our own minds’ propensities, it is at times useful to take
cognizance of the very different beliefs found in other cultures, which those
cultures consider perfectly reasonable. The Gurung tribesmen of Nepal spend a
lot of time climbing dangerously high trees to gather honey. When the French
photographer, Eric Valli, who worked extensively among the Gurung, asked them
if they do not ever fall down from their high perch, they replied: “Yes, you fall
when your life is over.” (National Geographic, vol. 193, no. 6, June 1998, p.92)
The above argument warns us that these “conclusions,” about causal links
and policy making can be wrong. By the use of examples and a simple argument, I
now show they are wrong.
Suppose researchers come to a town and do a randomized control trial on
the town population to check if the injection of a green chemical improves
memory and has adverse side effects. And suppose it is found that it has no side
effects and improves memory greatly in 95% of cases. If the study is properly
done and the random draw is truly random, it is likely to be treated as an
important finding and will, in all likelihood, get published in a major scientific
journal.
Now consider a particular woman called Eve who lives in this town and is
keen to enhance her memory. Can she, on the basis of this scientific study,
deduce that there is 0.95 probability that her memory will improve greatly if she
takes this injection? The answer is no, because she is not a random draw of an
individual from this town. All we do know from the law of large numbers is that
for every randomly drawn person from this population the probability that the
injection will enhance memory is 0.95. But this would not be true for a specially
chosen person in the same way that this would not be true of someone chosen
from another town or another time.
To see this more clearly, permit me to alter the scenario in a statistically-
neutral way. Suppose that what I called the town in the above example is actually
7
the Garden of Eden, which is inhabited by snakes and other similar creatures, and
Eve and Adam are the only human beings in this place. Suppose now the same
experiment was done in the Garden of Eden. That is, randomizers came, drew a
large random sample of creatures, and administered the green injection and got
the same result as described above. It works in 95% of cases. Clearly, Eve will have
little confidence, on the basis of this, to expect that this treatment will work on
her. I am assuming that the random draw of creatures on whom the injection was
tested did not include Eve and Adam. Eve will in all likelihood flee from anyone
trying to administer this injection to her, because she would have plainly seen
that what the RCT demonstrates is that it works in the case of snakes and other
such creatures; and the fact that she is part of the population from which the
random sample was drawn is in no way pertinent.
Indeed, and the importance of this will become evident later, suppose in a
neighboring garden, where all living creatures happen to be humans, there was a
biased-sample (meaning non-random 8) trial of this injection, and it was found that
the injection does not enhance memory and, in fact, gives a throbbing headache
in a large proportion of cases, it is likely that Eve would be tempted to go along
with this biased-sample study done on another population rather than the RCT
conducted on her own population in drawing conclusions about what the
injection might do to her. There is as little hard reason for Eve to reach this
conclusion as it would be for her to conclude that the RCT result in her own
Garden of Eden would work on her. I am merely pointing to a propensity of the
human mind whereby certain biased trials may appear more relevant to us than
certain perfectly controlled ones.
We shall return to this later. For now what is important to note is that even
if there is a result proved by a proper randomized controlled trial conducted on a
population to which you belong, you have no reason, on the basis purely of this
study, to believe that its finding will apply to you.
8
For instance, the trial was done early in the morning and so, inadvertently, only early risers were chosen for the
test.
8
The same reasoning carries over when one moves from a finding based on
one population to another population. The fact that the choice of a woman leader
of the village council in Bengal and Rajasthan led to the better provision of public
goods does not mean that you have reason to expect this will be true in Bihar,
Bahia or Bahrain, or, for that matter, in Bengal and Rajasthan tomorrow, because
tomorrow’s population is a different one. At least, if we were as fastidious about
deducing conclusions as the practitioners of randomized control trials want us to
be, we could not draw those conclusions about other places and other times. That
would be like drawing lessons from biased samples. This, in an interesting
converse way, is related to an important problem noted by Deaton (2010). Even if
we learn the facts from properly controlled randomized trials, if the intervention
is then to be made not on a random draw from the population but to a specially-
selected group, we should be prepared for failure. 9
3. A Limited Causality Claim
The claim that can be made, but after some prior assumptions are made, is
that randomized trials can establish certain ‘circumstantial’ causalities. The prior
assumption that we need is to accept the determinist claim that the universe
proceeds by causality and so the future that lies ahead of us is as determined as
our history. To state this more precisely, assume time progresses in pixels, instead
of a continuum, so that for each point of time, there is a well-defined immediate
previous point of time. Let X be the set of all possible states of the world at all
possible points of time. Further, X is a Cartesian product of n sets, X1, X2, … , Xn,
where each of these sets consists of all possible states of some particular feature
of the world. Thus X1 could consist of positions of the sun; X2 could be states of
the weather, etc. Hence, at any point of time, an x ε X is a full description of the
9
A stark example of this is provided by Worrall (2007). The drug, benoxaprofen, for arthritis and muscular skeletal
pain, passed the test of randomized trial very well. But when the drug was tried on actual patients it was found to
cause a disproportionate number of deaths. What later became clear is that this is a drug that is not used on a
random sample of people, but pre-dominantly on the older persons and they are the ones most likely to get an
adverse reaction. This powerful argument is also discussed and elaborated upon by Deaton (2010).
9
state of the world10. Of course, n will be a very large number; it is conceivable that
n is infinity.
In its minimal form, the determinist claim says that if at two points of time,
t and k, the world is in state x ε X, and at time t+1 it is in state y ε X, then at k+1 it
must also be in state y. In other words, being at x causes the world to be at y in
the next instant.
This simple axiom is untestable but has many implications. It means, for
instance, that if at two consecutive points of time the world is in the same state,
then onwards it will forever remain in that state. It follows that, since the world
thus far has been changing, it must have always been changing. This in itself does
not mean that the world never comes back to the same state. Put simply, it is not
that history never repeats itself but either history never repeats itself or it repeats
itself endlessly. The determinist axiom also has large implications for our attitude
to crime, responsibility and punishment, and much has been written by
philosophers on this. But that is not my concern here.
To see what randomized control trials show, let me define the Cartesian
product of X2, X3, … , Xn by Z. What RCTs show is that there exists some z ε Z, such
that if we have the world in state (x, z) ε X instead of (y, z) ε X, the world in the
next period will be in some a ε X instead of b ε X. This is like saying, other things
being the same (that is, z) if you vaccinate people, in the next period, there will be
no influenza. But if you did not vaccinate them, there would be influenza. If, we
accept the determinist axiom, as many do11, then this demonstration means that
whenever we switch from (y, z) to (x, z), the world will in the next period switch
from b to a. It is the ‘whenever’ that makes this a causal claim. And, this is what I
am referring to as ‘circumstantial causality’. Given a certain set of circumstances,
changing y to x, has a predictable consequence.
The discovery of circumstantial causal connections, as has happened with
the rise of RCT studies, is extremely valuable and, at the same time, of limited
consequence, more so than the proponents believe. As such, the RCTs have given
10
It should be clarified that the time when a state exists is not a feature of the state and so there cannot be a
reference to the temporal position of the state.
11
I personally do believe in the determinist axiom, while remaining prepared to be corrected (Basu, 2000).
10
us numerous valuable descriptions of what happened in the past and numerous
instances of causes in the past (provided of course that one is willing to accept the
determinist axiom). On the other hand, what they show is very limited. This is
because when they show that it was the switch from y to x that caused the switch
from b to a, what they are saying is that this was true under certain historical
conditions (z), but they cannot tell you what those historical conditions are. RCT
discoveries never graduate from something “was a cause” of something else to “is
a cause.” RCTs give us no insight into universal causality because it cannot tell us
what is it that was being held constant (z in the above example), when we
switched some intervention b to a. For Bengal in a certain period, electing a
woman leader of the local government caused water provisioning to be better.
This is no guide to the future, since we do not fully know what Bengal in a certain
period is like. Henceforth, a reference to causality without a qualifying epithet
should be taken to be a reference to universal causality, since for policy purposes
that is what is of essence.
One clarification is called for here. When we hold the z constant and
change y to x, in the above example, why are we saying that it is not known what
is being held constant and that we simply know that something is being held
constant? In the Chattopadhyay and Duflo (2004) example, for instance, that this
is true for Bengal of a certain period. Why is this not a useful description? The
reason is that this is a ‘dated description’—one in which the date or period of the
experiment is a part of the description of what is being held constant. Since a date
never recurs, we can never use this description in the future. So we will never be
able to say that the same conditions now hold and so the same result is to be
expected (recall the determinist axiom). What is minimally be needed is a
description of what is being held constant when a particular experiment is being
run without reference to a date. RCTs do not give this to us and this is what makes
the findings of RCTs non-portable.
Cartwright (2010) is right in her broad reservations about RCTs. However,
contrary to Cartwright’s claims, I am arguing that RCTs do provide insight into
circumstantial causality and is the gold standard for describing large populations
11
over time. On the other hand, they fail in demonstrating any form of universal
causality. They show us that, by the use of the law of large numbers, we can
describe the average characteristics of a large population and changes over time,
by appropriately studying a small sample drawn from the population. RCTs do this
extremely well.
In other words, what I have tried to demonstrate is a critique similar to that
of Cartwright (2010, 2011) but with a significant difference. On description and on
historical causal connections (without being able to show what needs to be held
constant for the causal link to work), RCTs are quite exemplary; but on universal
causality, which is the main claim of the practitioners of the method of
randomization and which is the crux of going from research to policy, RCTs get no
marks. On universal causal connections that tell us how to design policy
interventions the method of randomized controlled trials is silent. 12
4. Learning from Success and Failure
The advantage of randomized experiments in describing populations
creates an illusion of knowledge, which is not always easy to unravel. This
happens because of the propensity of scientific journals to value so-called causal
findings and not value findings where no (so-called) causality is found. In brief, it
is arguable that we know less than we think we do.
12
Causality in science and economics has long been the source of philosophical debate and skirmish: see, for
instance, Sen (1959), Hicks (1979), Hacking (1975), Skyrms (1980), Hoover (2001). In its purest form, the existence
of causality simply means that if there were two worlds which were identical just before time t, they cannot be
different at time t. If one buys into this as an axiom, as many prominent philosophers through the ages have done,
one adopts a 'deterministic' view of life. Among economists, Amartya Sen took a similar line in one of his earliest
writings—Sen (1959). I should hasten to add that though he has not directly written on this subsequently, there is
some smoking-gun evidence from his other writings that his views on this have changed.
It may appear at first sight that this pristine form of causality does not have any implication for what we
do and how we behave and so would be dismissed at least by logical positivists as a debate of no consequence.
This is however not right since whether or not one is a determinist has important implications concerning the
moral responsibility of individuals for their actions, with implications for systems of justice and the punishment
that is meted out to those whose actions are found unacceptable to society.
12
To see this, suppose--as is indeed the case in reality--that thousands of
researchers in thousands of places are conducting experiments to reveal some
causal link. Let us in particular suppose that there are numerous researchers in
numerous villages doing randomized experiments to see if M causes P. Words
being more transparent than symbols, let us assume they want to see if medicine
(M) improves the school participation (P) of school-going children. In each village,
ten randomly-selected children are administered M and the school participation
rates of those children and also children who were not given M are monitored.
Suppose children without M go to school half the time and are out of school the
other half. The question is: Is there a systematic difference of behavior among
children given M?
I shall now deliberately construct an underlying model whereby there will
be no causal link between M and P. Suppose Nature does the following. For each
child, whether or not the child has had M, Nature tosses a coin. If it comes out
tails the child does not go to school and if it comes out heads, the child goes to
school regularly.
Consider a village and an RCT researcher in the village. What is the
probability, p, that she will find that all ten children given M will go to school
regularly? The answer is clearly
p = (1/2)10
since we have to get heads for each of the ten tosses for the ten children.
Now consider n researchers in n villages. What is the probability that all n
villages will have mixed outcomes, by which I mean, some children will participate
in school and some will not (i.e., some of the ten tosses in each village will result
in heads and some in tails)? Clearly, the answer is: (1 – p)n.
Hence, if ϕ(n) is used to denote the probability that among the n villages
where the experiment is done, there is at least one village where all ten tosses
come out heads, we have:
13
ϕ(n) = 1 – (1 – p)n.
Check now that if n = 1, that is, there is only one village where this
experiment is done, the probability that all ten children administered M will
participate in school regularly is ϕ(1) = 0.001. In other words, the likelihood is
negligible.
It is easy to check the following are true:
ϕ(100) = 0.0931
ϕ(1,000) = 0.6236
ϕ(10,000) = 0.9999
And therein lies the catch. If the experiment is done in 100 villages, the
probability that there exists at least one village in which all tosses result in heads
is still very small, less than 0.1. But if there are 1,000 experimenters in 1,000
villages doing this, the probability that there will exist one village where it will be
found that all ten children administered M will participate regularly in school is
0.6236. That is, it is more likely that such a village will exist than not. And, if the
experiment is done in 10,000 villages, the probability of there being one village
where M always leads to P is a virtual certainty (0.9999).
This is, of course, a specific example. But that this problem will invariably
arise follows from the fact that
limn→∞ ϕ(n) = 1 – (1 – p)n = 1.
Given that those who find such a compelling link between M and P will be
able to publish their paper and others will not, we will get the impression that a
true causal link has been found, though in this case (since we know the underlying
process) we know that that is not the case. With 10,000 experiments it is close to
certainty that someone will find a firm link between M and P. Hence, the finding
of such a link shows nothing but the laws of probability being intact. Yet, thanks
to the propensity of journals to publish the presence rather than the absence of
14
“causal” links, we get an illusion of knowledge and discovery where there are
none.
One practical implication of this observation is that it spells out the urgent
need for a Journal of Failed Experiments or at least a publicly-available
depository of such experiments so that we know how many people found a
relation between M and P and how many did not. This will not solve all problems,
since arguably, in each of the villages where researchers were looking for the
relation between M and P, it is possible that M and P stood for different things
from what others were looking for in other villages. If each underlying process
entails the toss of a coin as in the above description, then we will not have
evidence on the same test in many places, but different tests in different places.
Nevertheless, it is important to keep track of failed experiments, in the sense of
experiments that did not reveal any link between two variables. And such a
journal, there can be little doubt, will have a sobering effect on economics,
making evident where the presence of a result is likely to be a pure statistical
artifact.
5. Knowledge, Evolution and the Induction Principle
Despite the skepticism of the previous sections, the fact remains that
human beings know a lot. We “know” when we release a ball in mid-air, it will
move down; we “know” what time the sun will rise tomorrow; we “know” that it
is a virus that is the source of the common cold. Of course, all this can be the luck
of the draw, implying that all this supposed knowledge worked thus far, but will
not tomorrow. This cannot be ruled out formally, but intuition suggests there is
more to these bits of knowledge than pure luck. At any rate, that is the
presumption under which I will work. Human beings know a lot about nature or,
more minimally, our minds are somehow synchronized with nature. We tend to
observe patterns in nature, where indeed there are patterns. And most of what
our mind knows it does simply by picking up knowledge along the way without
pausing to think if the knowledge has scientific basis and if it will stand the test of
randomization.
15
A child soon learns that a person rolling over with laughter is happy, the
woman weeping quietly in a corner is sad, the man with ruffled hair running
toward another man with an open knife is likely to do harm. If we acted as
custodians of knowledge by randomization, and stopped the child at each stage
and asked the child to reject these pieces of knowledge until they had verified
them by conducting RCTs, that is by making sure that randomly-selected persons
rolling over with laughter were happier than others, and so on, then it is likely
that the child’s cognitive abilities would be deeply damaged.
If we truly paused to think, we would recognize that the proportion of
knowledge that we informally absorb is substantially greater than what we know
based on scientific enquiries. To dismiss the former out of hand would greatly
deplete our knowledge.
As the example of the Garden of Eden, above, shows, in acquiring
knowledge there is scope and indeed need for a catholicity of methods, since our
intuition can help fill the gaps. A recent study (Paul and Dredze, 2011) shows that
there is an amazingly strong correlation between the incidence of flu in the US
and the extent of chatter related to the flu on twitter. Seeing this, most of us will
be prone to assuming that it is not the chatter that causes the flu. And this would
seem to me to be the right conclusion, and there would be no reason to insist
that this be established by conducting proper randomized trials. 13
But how do children pick up knowledge from such ascientific processes?
While there is no clear answer, one incomplete answer relates to evolution. Only
those human minds that are in reasonable synchrony with nature and can glean
knowledge from experience have survival value. Hence, over long stretches of
13
One can go further and argue that even without correlation, the pure act of measurement facilitates
understanding. Gates (2013) gives numerous examples, though without going into why measurement works. It is
arguable that measurement makes it easy for our minds to process information and develop and intuitive and
heuristic understanding, the role of which must not be minimized (Gigerenzer and Gaismaier, 2011; Gilovich,
Griffin and Kahneman, 2002).
16
time, minds with this ability have been selected; and we, at the start of the
twenty-first century, are lucky to be endowed with such ability. 14
What will be argued here is that we have no choice but to use this faculty of
ours to go from experience—both statistically sound ones based on RCTs and also
ones based on biased samples—to the crafting of policy. There is actually no
other option. As we saw in the previous sections, RCTs do not in themselves tell
us anything about the traits of populations in other places and at other times.
Hence, no matter how large the population is from which we draw our random
samples, since it is impossible to draw samples from tomorrow’s population and
all policies we craft today are for use tomorrow, there is no “scientific” way to go
from RCTs to policy. En route from evidence and experience to policy, we have to
rely on intuition, common sense and judgment. 15 It is evidence coupled with
intuition and judgment that gives us knowledge. To deny any role to intuition is to
fall into total nihilism. 16
Some critics have pointed out that the method of experiments and random
trials deflects us from the important aim of explaining “why” certain regularities
in nature occur instead of merely finding such regularities. This is an important
observation made by very prominent commentators in this debate (see, for
instance, Deaton, 2010; Heckman and Smith, 1995) and answering to the “why” is
indeed an important ingredient of human understanding. However, I would argue
that as a critique of RCTs, this is invalid (see also McKenzie, 2012). To see this, let
me point out, as will become evident later, that some of our so-called
understanding is delusional. It is based on the mind observing regularities and,
after some time, treating these regularities as facts of nature. Then when we see
14
I tried to develop this idea in Basu (2000), utilizing the prior work of Lorenz (1977).
15
As I venture into this controversial territory, it is useful to marshal some support from Leonard Savage: “I believe
that here, as elsewhere, catastrophe is avoided, primarily because in practical situations commonsense generally
saves all but the most pedantic of us from flagrant error.” (Savage, 1954, p. 1).
16
Total nihilism regarding knowledge is not a line to be dismissed out of hand and several prominent philosophers
have taken such a line and there are probably even more philosophers who have taken such a line than we know
since it is reasonable to expect that many of them would not write anything and maybe not even speak about it.
17
something new happen and say that we have understood it, it is because we have
managed to fit this new observation into the partly solved jigsaw puzzle where
the other parts are provided by our previous observation of regularity and our
propensity to treat those as immutable parts of nature. To the extent that RCTs
feed our minds with more and more observed and statistically verified
regularities, they actually help with our understanding of why certain things
happen, instead of deflecting us from such an understanding. So this arrow,
despite having been shot by such shrewd observers, does not slay the claims
made on behalf of the method of randomization. But there are other arrows.
The first ingredient for going from past regularities to expectations about
the future is the “induction principle.” Broadly, this says that if we see something
that has happened repeatedly in the past, like the sun having risen every day, we
have reason to expect this will happen in the future. We have no escape from
using the induction principle in acquiring (falsifiable) knowledge. 17 That qualifier,
“falsifiable,” is necessary because there are certain kinds of deductive knowledge
that do not require any evidence or data. This is true of, for instance, the
Pythagoras theorem on right-angled triangles, or Arrow’s Impossibility theorem
on voting. The knowledge of these results does not require facts and so is
independent of the principle. But, in this paper, I am keeping such non-falsifiable
knowledge aside and focusing attention exclusively on knowledge which is, in
principle, falsifiable.
The problem with the induction principle, so essential for this kind of
knowledge, is that it is inherently fuzzy; and it is possible for intelligent people not
to accept it. This stems from the fact that the validity of the induction principle
depends on our priors, and priors, by their very nature, cannot be proved. 18
17
Philosophers have long battled with analyzing and trying to understand induction (for instance, Mill, 1843;
Bunge, 1959; Skyrms, 1840). For an historical account, see Hacking (1975).
18
Moreover, as John Stuart Mill had noted in the mid-nineteenth century (1843), the induction principle, unlike
deduction, does not lead us to apodictic certainty.
18
Let us see what kinds of priors get us to the induction principle. Suppose
you have the prior that nature sits with an urn containing a very large number of
balls which could be (1) all white, (2) all black and (3) some white and some black.
You believe that just before dawn nature picks a ball. If it is white, the sun is made
to rise and if it is black, the sun is stopped from rising. Now consider your first
dawn. If the sun rises, you know (2) cannot be valid. Then as the sun continues to
rise every morning, you gradually begin to rule out (3) and base your expectations
on (1). The more ample the evidence is, the harder your belief becomes that the
sun will rise tomorrow as it has done in the past. 19
But consider a person with different priors. She believes either (2) or (3),
above, is true. Now when the sun rises the first day, (2) is ruled out. Then as the
sun rises each morning, her prediction with every passing day will go the other
way around. She will become more and more firm in the expectation that the sun
will not rise tomorrow. 20 She will reject the induction principle and there is no
way to persuade her that she is wrong. It is my prior versus hers. 21
19
Russell (1912), in an essay on induction, has made this very clear with the example of the chicken trained in the
art of induction who expects to have its neck patted every day only to have it wrung one fine morning. This risk will
always be there but one can amass greater and greater amount of data from the past to provide strength to the
inductive process. Thus we may have begun by arguing that the sun will rise tomorrow because it has done so in
the past. But if we could base this on the larger body of evidence that we now have about spinning bodies
continuing to spin, we will have an even longer data base to base our induction on. In some sense, this is the heart
of the scientific enterprise.
20
If her prior is that the urn contains a large and finite number of balls, will not be able to compute the probability
that the sun will not rise tomorrow but with will be sure with each passing day that the probability of the sun not
rising will become larger. If the ball has z balls and w of those are white, then after the sun has risen for y days, her
estimated probability of the sun not rising the next day will be η ≡ (z - w)/(z – y). Clearly, as y increases, so does η.
21
And there are many examples where a blind adherence to the induction principle does not work. Folklore
attributes the example that follows to Galileo. Modernizing the example a little, suppose a man, rooted to the
induction principle and impatient about wasting time understanding calendars, asks his neighbor each morning, “Is
st
today 1 November 2013?” and gets the answer “no.” This continues for days, months and years and he finally
st
predicts: “No morning will be 1 November 2013.” But of course he will, as I pointed out in my Albert Hirschman
st
lecture delivered on 31 October 2013, “get a big surprise tomorrow.”
This is a light-hearted argument but cannot be dismissed out of hand. This along with the argument alerts
us to the fact that science, which is so rooted in induction, should be taken with a dose of skepticism. This is my
one argument with Dawkins (2006). The skepticism he expresses about religion and faith is right; but he falls into
some of the same traps he warns us against when it comes to science. The human proclivity for superstition does
not end with religion.
19
The induction principle cannot of its own reach definite conclusions. As
Manski (2013, p. 30-31) observes, “The logic of inference does not enable any
conclusions about future or hypothetical situations to be drawn based on
observed tendencies per se. Assumptions are essential.” This inherent ambiguity
of the induction principle and the need for tempering it with assumptions or
human judgment opens us to the risk of “anything goes.” Human beings have
often been subject to blind faith and strange delusions of knowledge and
superstitions and large mistakes in decisions have been made and continue to be
made because of this. Hence, it needs to be emphasized that the method of using
the best available evidence and intuition to derive policy has to guard against this
risk. Just as we have intuition, we also have reason. While, as I have tried to show,
there is no getting away from the use of intuition, what we need to do is to
harness reason to vet our intuition. What we have to rely on is best described as
reasoned intuition.
6. Reasoned Intuition
Making good policy entails the use of the best available evidence. But to do
this right we need reason and that is often a stumbling block. Moreover, the fact
that we should use the best available evidence does not imply that policies must
invariably be based on evidence. To see this, consider the common criticism
whenever someone proposes a new policy. Assume that a person recommends a
new policy intervention, called X, without providing any evidence on whether or
not X works. X could be a new way to make conditional cash transfers to the poor.
It is common in such a situation to find critics who will oppose this policy X on the
ground that there is no evidence whether or not X works. What these critics do
not realize is that such a criticism is self-contradictory and thus invalid.
To understand this, let “not doing X” be called “doing Y.” Clearly, if there is
no evidence whether or not X works, there is no evidence whether or not Y works.
If it is wrong to implement X if there is no evidence whether or not X works, it is
wrong to implement Y if there is no evidence whether or not Y works. But since Y
20
is the negation of X, it is meaningless to say that we should not implement X or Y.
This contradiction establishes that the initial criticism must be invalid.
This is simply a cautionary tale about the ubiquity of unreason and the
errors it leads us into. Of course, the best available evidence must be used, and
we must learn from past patterns and regularities about what to expect in the
future. What is often not recognized is that when people try to follow the widely-
used dictum, “policies should be evidence based,” the hardest part of the rule is
contained in the word “based.” This is the reason why we so often hear glib
references to some evidence and then a hand-waving switch to one’s favored
policy, ignoring the fact that the policy should be based on the best available
evidence.
But how do we make this transition from evidence to policy? As I argued
earlier, our intuition has something innately valuable in it. This may well be an
outcome of evolution. If the human intuition were completely out of line with
nature and read patterns wrong, presumably it would have been weeded out by
the process of natural selection. However, as the above example shows, our pure
intuition and gut feeling may not be quite flawless. They need to be held under
the scanner of reason before we use them to translate experience and evidence
into rules of behavior and policy. I use the expression “reasoned intuition” to
refer to intuition after it has been subjected to thought and scrutiny by the other
faculty that we all possess, namely, reason. It is not possible to give “reasoned
intuition” a hard definition. Its importance is founded on the realization that our
intuitions are not flawless. Major blunders have been committed in the past by
humankind on the basis of intuition and a refusal to subject intuition to scrutiny.
Fortunately, we also possess the ability to reason. What I am arguing is that
by bringing these two faculties—intuition and reason—together, we can greatly
improve our ability to make policies. Is my intuition consistent with the evidence
we have and the other things I already know? Is it consistent with the other
intuitive beliefs I hold? Reasoned intuition is intuition that has been subjected to
this kind of interrogation.
21
Indeed, a large part of the present paper tried to show that there is no
getting away from the use of reasoned intuition. Empirical studies, no matter how
carefully done, cannot in themselves lead us to particular policies. Going from
evidence, data, and statistics to policy will invariably entail a leap of imagination.
This is facilitated by our intuition, honed over millennia by the processes of
evolution. But we must stand ready to subject the intuition to reason. This cannot
be stated as a hard rule; but to ignore it on that ground would be a mistake.
I can illustrate some kinds of common mistakes that our intuition often
leads us to. The remedy is not to discard the use of intuition but to pause and
check that it stands the test of reason. One empirical finding which has been at
the base of a lot of development policy discussion is the role of growth in
mitigating poverty. Dollar, Kleineberg and Kraay (2013) have recently shown,
using a remarkably comprehensive data set, spanning multiple countries and
several decades, that the bulk of poverty reduction that the world has seen in the
past was due to overall GDP growth. Intuition then prompts us to propose that we
should therefore rely on growth rather than specially designed market
interventions to battle poverty. For the untutored, the growth v. poverty debate
is an emotive one and this observation quickly gathers support among those who
have a predilection to leave it all to the market. Without going into this larger
debate here (see Basu, 2013, for such a discussion), let me simply point to an
obvious mistake that leads us to make this policy conclusion from this robust
empirical finding. The fact that 75% of the drop in poverty in the past occurred
because of growth in no way shows that growth is more effective in eradicating
poverty than say a new conditional cash transfer program. The mistake in making
that deduction would be the same as someone studying infections in the 1930s
asserting that we should rely on non-penicillin medicines because past data
shows that 99% of all cures were because of non-penicillin drugs. This ignores the
fact that penicillin was discovered in 1928 and so the lack of evidence of the
success of penicillin is not a sign of penicillin not working but of penicillin not
existing.
22
In other words, if we want to use the induction principle to decide if policy
X is likely to achieve a certain result R, it is not enough to show that R was
achieved in the past because of Y, and from this conclude X will not work. It is
important to check if X was tried in the past and then make the deduction. In case
X never achieved R because X was never tried, we do not have evidence on the
efficacy of X one way or the other. To assume otherwise is to make the same
mistake as the one discussed at the start of this section, whereby the lack of
evidence in favor and against a new policy is taken to be a verdict against the new
policy. As was pointed out earlier, the induction principle does not have a precise
definition; but, upon cogitation, we can nevertheless conclude that supposedly
inductive conclusions are wrong. It is this which makes the inductive principle
useful.
The mistake about growth and poverty just discussed is the same one that
we so often hear being used for job creation. People often assert that we have to
rely on the private sector for job creation because past data show that 80% of
jobs were created by the private sector. If this logic were correct, we would also
have to accept the logic of an economist in Soviet Russia in 1980 arguing that we
have to rely on the state for job creation because past data show that 90% of all
jobs were created by the state.
These are all examples of using our intuition wrongly to go from induction
to certain policy prescriptions. Since the induction principle is not one that can be
formally defined, mistakes are also not always possible to formally define. Yet, if
we pause to reason carefully or listen to someone else provide the reasoning, we
would, on our own, back off from many a wrong policy. This is what reasoned
intuition is all about.
What is the role of theory in all this? It is common to pay lip service to this
and assert that we need both theory and data to make policy prescriptions. There
are many instances where we do not need any theory. Spotting regularities in
data, coupled with reasoned intuition, can take us to useful policy prescriptions.
23
There can be no need for theory. Likewise, there are areas where theory, coupled
with what we intuitively know, can take us to useful policy decisions.
The real role of theory is to make consistency checks on our intuitive
beliefs22 and, based on our intuitive beliefs, to take steps toward a deeper
understanding using pure deduction. Can all the beliefs we hold be together
valid? Theory is a method for checking this. It also helps us take deductive steps in
conjunction with what we know, whether through formal empirical studies or
simply through the informal acquisition of knowledge. The role of theory and
empirics is best understood with an analogy with Sudoku. Suppose we have a
slightly different Sudoku game in which the numbers that are typically entered in
the squares to start with are not shown explicitly but have to be found out using
clues (“look under a pillow for the number that goes into the left-hand square).
The search for these preexisting but hidden numbers is akin to empirical
investigation. However, once we have found out all the numbers specified in
advance, the rest of the Sudoku proceeds by pure reasoning. All the other
numbers have to be deduced rather than found. This is akin to what theory does.
It enables us to move on using the given facts and by using pure deduction.
7. Concluding Remarks
The rise of the method of randomization has been a major stimulus to
empirical development economics and as such it is a welcome advance. Akin to
what this method did earlier for epidemiology, it has in its short life in economics
brought to light several new facts that were earlier languishing unseen. This
deserves credit and further use. However, the claims made by its proponents
concerning its ability to reveal causal links that directly lead to policy conclusions
are wrong. That is what this paper tried to argue. RCTs have no advantage on
these scores. What RCTs do usefully and well is to describe. They give us ways to
describe static features and also temporal features (M happened in period 1 for
22
As Rubinstein (2012, p. 129) observes, “My view of game theory is consistent with my approach to economic
models in general … . Game theory does not try to describe reality or be normative. Game theory investigates the
logic of strategic thinking.”
24
some members of the population and P happened in period 2 for 90% of those to
whom M happened and 10% of those to whom M did not happen) with no clues
to universal causality. They can reveal circumstantial causality, namely, that, in
certain historical circumstances, doing x instead of y, led to a instead of b, in the
next period, but since we do not have a full description of the historical
circumstances, there is, strictly, no lesson in this which is portable through time.
Universal causality is a mental construct for observers, something that
helps them apply intuition and judgment, in conjunction with what they know or
think they know, to decide how to behave and which policies to choose. What
human beings know comes from many sources, and to deem only one method
valid and all others invalid is to slow the process of knowledge acquisition. The
catholicity of methods currently used—from anthropological notes, analysis of
large data sets, everyday experience and randomized trials—all have a role to play
in this enterprise. When it comes to seeking good descriptions of past facts, RCTs
constitute the best method to aspire to. When it comes to the search for
universal causality and best policy, there is a role for multiple sources, including
reasoned intuition, honed over millennia of human evolution. And all this should
be tempered with a shot of skepticism—an awareness that for all our best efforts,
we may be wrong.
25
References
Ashraf, Nava, Karlan, Dean and Yin, Wesley (2006), ‘Tying Odysseus to the Mast:
Evidence from a Commitment Savings Product in the Philippines,’ Quarterly
Journal of Economics, vol. 121.
Banerjee, Abhijit (2005), ‘‘New Development Economics’ and the Challenge to
Theory,’ Economic and Political Weekly, October 1.
Banerjee, Abhijit and Duflo, Esther (2009), ‘The Experimental Approach to
Development Economics,’ Annual Review of Economics, vol. 1.
Banerjee, Abhijit and Duflo, Esther (2012), Poor Economics: A Radical Rethinking
of the Way to Fight Poverty, Public Affairs.
Bardhan, Pranab (2013), ‘Two Ideas about Fighting Global Poverty,’ Boston
Review, May/June.
Barrett, Christopher and Carter, Michael (2010), ‘The Power and Pitfalls of
Experiments in Development Economics: Some Non-Random Reflections,’
Applied Economic Perspectives and Policy, vol. 32.
Basu, Kaushik (2000), Prelude to Political Economy: A Study of the Social and
Political Foundations of Economics, Oxford University Press, Oxford.
Basu, Kaushik (2005), ‘New Empirical Development Economics: Remarks on Its
Philosophical Foundations,’ Economic and Political Weekly, vol. 40,
October 1.
Basu, Kaushik (2011), Beyond the Invisible Hand: Groundwork for a New
Economics, Princeton University Press, Princeton, and Penguin, New Delhi.
Basu, Kaushik (2013), ‘Shared Prosperity and the Mitigation of Poverty: In Practice
and in Precept,’ World Bank Working Paper.
http://documents.worldbank.org/curated/en/2013/11/18506691/shared-
prosperity-mitigation-poverty-practice-precept
Beamen, Lori, Chattopadhyay, Raghav, Duflo, Esther, Pande, Rohini and Topalova,
Petia (2012), ‘Powerful Women: Does Exposure Reduce Bias?’ Quarterly
Journal of Economics, vol. 124.
Bernard, Tanguy, Dercon, Stefan, Orkin, Katie, and Taffesse, Alemayehu (2013),
26
‘Learning with Others: A Field Experiment on the Formations of Aspirations
in Ethiopia,’ mimeo: Oxford University.
Bruhn, Miriam and McKenzie, David (2009), ‘In Pursuit of Balance: Randomization
in Practice in Development Field Experiments,’ American Economic
Journal: Applied Economics, vol. 9.
Bunge, Mario (1959), Causality and Modern Science, 4th Edition, 2009,
Transaction Publishers, New Brunswick.
Cartwright, Nancy (2007), Hunting Causes and Using Them: Approaches to
Philosophy and Economics, Cambridge University Press, Cambridge.
Cartwright, Nancy (2010), ‘What Are Randomized Trials Good For?’ Philosophical
Studies, vol. 147.
Cartwright, Nancy (2011), ‘The Art of Medicine: A Philosopher’s View of the Long
Road from RCTs to Effectiveness,’ The Lancet, vol. 377, April 23.
Cartwright, Nancy and Hardie, Jeremy (2012), Evidence Based Policy: A Practical
Guide to Doing It Better, Oxford University Press, Oxford.
Chattopadhyay, Raghabendra and Duflo, Esther (2004), “Women as Policy
Makers: Evidence from a Randomized Policy Experiment in India,”
Econometrica, Vol. 72(5).
Das, Jishnu, Devarajan, Shanta, and Hammer, Jeffrey (2011), ‘The Welfare
Economics of Behavioral Economics and Randomized Trials,’ World Bank
Blog.
http://blogs.worldbank.org/files/africacan/DDH_The%20Welfare%20Econo
mics%20of%20Behavioral%20Economics%20and%20Randomized%20Contr
ol%20Trials(1).pdf
Dawkins, Richard (2006) The God Delusion, Bantam Books, London.
Deaton, Angus (2009) “Instruments of development: Randomization in the tropics
and the search for the elusive keys to development,” Keynes Lecture,
British Academy, Proceedings of the British Academy 2008 Lectures, 162,
Oxford University Press, Oxford.
Deaton, Angus (2010), ‘Instruments, Randomization, and Learning about
Development,’ Journal of Economic Literature, vol. 48.
27
Dollar, David, Tatjana Kleineberg and Kraay, Aart (2013) “Growth Still Is Good for
the Poor,” World Bank Policy Research Working Paper 6568. World Bank:
Washington D.C.
Duflo, Esther, Glennerster, Rachael and Kremer, Michael (2008), ‘Using
Randomization in Development Economics Research: A Toolkit,’ in T. Paul
Schultz and J. Strauss (eds.), Handbook of Development Economics, vol. 4,
Elsevier.
Elbers, Chris and Gunning, Jan (2013), ‘Evaluation of Development Programs:
Randomized Controlled Trials or Regressions?’ World Bank Economic
Review, vol. 27.
Gates, Bill (2013), ‘Measuring Progress,’ in Annual Letter, Gates Foundation.
http://annualletter.gatesfoundation.org/#nav=section1&slide=0
Gilovich, Tom, Griffin, Dale and Kahneman , Daniel (2002), Heuristics and Biases:
The Psychology of Intuitive Judgment, Cambridge University Press,
Cambridge.
Gigerenzer, Gerd and Gaissmaier, Wolfgang (2011), ‘Heuristic Decision Making,’
Annual Review of Psychology, vol. 62.
Habriyamana, James and Jack, William (2012), ‘State vs. Consumer Regulation: An
Evaluation of Two Road Safety Interventions in Kenya,’ NBER Working
Paper.
Hacking, Ian (1975), The Emergence of Probability: A Philosophical Study of Early
Ideas about Probability, Induction and Statistical Inference, Cambridge
University Press.
Hacking, Ian (1988), ‘Telepathy: Origins of Randomization in Experimental Design,’
Isis, vol. 79, No. 3.
Heckman, James and Smith, Jeffrey (1995), ‘Assessing the Case for Social
Experiments,’ Journal of Economic Perspectives, vol. 9.
Hicks, John (1979), Causality in Economics, Clarendon Press, Oxford.
Hoff, Karla, and Pandey, Priyanka (2006), ‘Discrimination, Social Identity, and
Durable Inequalities,’ American Economic Review, vol. 96.
Hoover, Kevin (2001), Causality in Macroeconomics, Cambridge University Press,
Cambridge.
Ifcher, John and Zarghamee, Homa, (2011), ‘Happiness and Time Preference: The
28
Effect of Positive Affect in a Random Assignment Experiment,’ American
Economic Review, vol. 101.
Iyer, Lakshmi, Mani, Anandi, Mishra, Prachi and Topalova, Petia (2012), ‘The
Power of Political Voice: Political Representation and Crimes against
Women in India,’ American Economic Journal: Applied Economics, vol. 3.
Jensen, Robert (2012), ‘Do Labor Market Opportunities Affect Young Women’s
Work and Family Decisions?’ Quarterly Journal of Economics, vol. 127.
Lorenz, K. (1977), Behind the Mirror: A Search for the Natural History of Human
Knowledge, (translated from German by R. Taylor), Harcourt Brace and
Jovanovich, New York.
Manski, Charles (2013), Public Policy in an Uncertain World, Harvard University
Press, Cambridge, MA.
McKenzie, David (2012), ‘Does Angus Deaton Worry Too Much about Wolves
Eating His t-Values?’ World Bank Blog.
http://blogs.worldbank.org/impactevaluations/does-angus-deaton-worry-
too-much-about-wolves-eating-his-t-values
Miguel, E. and Kremer, M. (2004), ‘Worms: Identifying Impacts on Education and
Health in the Presence of Treatment Externalities,’ Econometrica, vol.72.
Mill, John Stuart (1843), A System of Logic, Ratiocinative and Inductive.
Cambridge Books Online edition 2011.
Mookherjee, Dilip (2005), ‘Is There Too Little Theory in Development Economics
Today? Economic and Political Weekly, vol. 40, October 1.
Paul, Michael and Dredze, Mark (2011), ‘You Are What You Tweet: Analyzing
Twitter for Public Health,’ Association for the Advancement of Artificial
Intelligence.
Ravallion, Martin (2009), ‘Should the Randomistas Rule?’ Economists’ Voice.
http://siteresources.worldbank.org/INTPOVRES/Resources/477227-
1142020443961/2311843-
1229023430572/Should_the_randomistas_rule.pdf
Richet, Charles (1884), ‘La suggestion mentale et le calcul des probabilities,’
Revue philosophique de la France et de l’etranger, vol. 18.
Rodrik, Dani (2008), ‘The New Development Economics: We Shall Experiment but
How Shall We Learn?’ in J. Cohen and W. Easterly (eds.), What Works in
29
Development? Thinking Big and Thinking Small, Brookings Institution
Press, Washington, DC, 2009.
Rubinstein, Ariel (2012), Economic Fables, Open Book Publishers.
Russell, Bertrand (1912), The Problems of Philosophy, Re-published by Project
Gutenberg, 2009.
Savage, Leonard (1954), The Foundations of Statistics, John Wiley & Sons. (Page
references are to the revised Dover publication, 1972.)
Sen, Amartya (1959), ‘Determinism and Historical Predictions,’ Enquiry, vol. 2.
Sen, Amartya (1980), ‘Description as Choice,’ Oxford Economic Papers
Skyrms, Brian (1980), Causal Necessity, Yale University Press, New Haven.
Worrall, John (2007), ‘Evidence in Medicine and Evidence-Based Medicine,’
Philosophy Compass, vol. 2/6.
30