5
Key constraints Cannot interfere significantly with program operations Cannot be perceived as unfair Must be politically feasible Must be ethical

6
One Trick: Limited Resources Many programs only have so much resources Many more people are eligible than they could serve Examples: –Training program –Limited textbooks to distribute How to allocate resources?

7
Lottery design Randomly choose from applicant pool In this design participants are explicitly told that “winners” and “losers” will end up with different outcomes Use when there is no good reason to discriminate among a subset of applicants Lottery is perceived as fair Transparent Politically more feasible often

9
Lottery Design Lottery need not be over individuals Can be over groups Which micro-credit groups? Can be over communities Which villages should we enter? Can be over schools Balsakhi

10
A lottery over program sites Works if there are multiple sites How are the program sites chosen? Some choices arbitrary? What is the affected population?

11
Ethicality and Political Feasibility Transparency often helps Can we give something to the control sites? What happened in the balsakhi intervention?

12
povertyactionlab.org What to do if program has 500 applicants for roughly 500 slots? Or less?

13
A less obvious lottery design One trick: Advertise program and greatly increase applicants –Now lottery is feasible Is this ethical? Would it be ethical we added slots to the program and only gave these to the new applicants? Or desirable from the evaluation point of view?

17
What can be done? Potential Solution: –What are they screening for? –What elements are essential? –What elements are arbitrary? Example: Training program –2000 candidates –1000 fit criteria such as poor enough to be worthy, qualified enough to take advantage of training –250 chosen of out of the 1000 based on particular attributes –NGO willing to allocate remaining 250 by lottery Lesson: Many parts of a selection rule are designed simply to weed out candidates

18
povertyactionlab.org Can we think of any other trick that will work here?

19
Phase-in Design In this design everyone is told that they will end up with the same outcome but some later than others. Use the fact that the program going to expand Example: –In 5 years, 500 schools will be covered. What determines which schools will be covered? –Some choices may be based on need, potential impact, etc. –Some choices largely arbitrary We can therefore choose who gets program early at random –Do this on population about which choices are arbitrary What is the affected population? How are the comparisons made?

21
Encouragement design In this design everyone is immediately eligible to receive the program—there is enough funding to cover all of them. However not all of them will necessarily take it up Pick some people at random and encourage them to use program Use non-encouraged as control Ethical? What population is this a treatment effect on?

26
Individual versus collective randomization Pros of individual level randomization: –Data collection is easier: counterexample? –We can measure the impact of individual characteristics on what the program does Cons: –The program may naturally operate at another level In the balsakhi case how would we do an individual level randomization? –May create conflict within a group/community –Even if the treatment is on the individual the effect may go beyond him.

27
Randomize evenly? Suppose 100 slots are available in a training program. Two hundred men and women apply You are able to randomize these 100 slots One strategy: –Give each person a lottery number 1 to 200. –Randomly pick 100 numbers. –Those people are chosen Anything wrong with this strategy?

28
A Simplified India For simplification, we reference only the sixteen largest Indian states These 16 states are divided into four geographic regions We also assume there are exactly 20 districts in each Indian state, implying 320 districts in all of India.

30
Experiment Design For Program X, we plan to run an intervention with 40 treatment districts and 40 control districts. These districts are randomly selected from the entire pool of 320 Indian districts. The composition of the treatment group, broken down by geographic region, varies considerably each time we randomize the sample.

34
Stratification Divide India into four regions of 80 districts each. Randomly choose 10 control and 10 treatment in each. Why not 70 in control?

35
Problems with unbalanced treatment and control groups Shocks hitting some regions but not others. Suppose you wanted to know the treatment effect in each region?

36
Advantages of Stratification Good control groups. Ensure that each area is represented in treatment and control –Important if some shocks are hitting men or women specifically Allows estimation of treatment effects for each group –Important if you care about differential treatment for men and women

37
povertyactionlab.org Would estimates be biased if there were no stratification? No. Stratification increases power

38
Variables to stratify on Context specific Groups affected by different shocks –Different regions –Different income groups –Different occupations Groups with different treatment effects What happened in the balsakhi case?