Center for Survey Research University of Virginia Center for Survey Research University of Virginia 1 Survey Experiments: Past, Present, Future Thomas.

Similar presentations

Presentation on theme: "Center for Survey Research University of Virginia Center for Survey Research University of Virginia 1 Survey Experiments: Past, Present, Future Thomas."— Presentation transcript:

1
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 1 Survey Experiments: Past, Present, Future Thomas M. Guterbock Director, Center for Survey Research University of Virginia 2010 Annual Conference Harvard Program in Survey Research October 22, 2010

2
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 2 Overview Why survey experiments are so cool Defining the survey experiment Methods vs. substance Scan of survey methods experiments Substantive experiments Key design issues in survey experiments Factorial (“vignette”) surveys Example: dirty bomb scenarios in the National Capital Region A look at the future of survey experiments

4
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 4 Two knowledge gaps Psych experimenters don’t always know a lot about doing surveys –Some don’t think sampling is very important –They don’t think surveys measure things well Survey researchers don’t always know a lot about experiments –And they question the external validity or relevance of many lab experiments Assumption: this audience, like the author, is more likely to be in the latter group

5
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 5 What’s a survey experiment? (and what’s not)

6
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 6 Experiments generally An intervention, treatment or stimulus is varied –Subjects randomly assigned to treatment vs. control Outcomes are measured Because of random assignment, any variation in outcomes can be attributed to the treatment –Absent various threats to internal validity The ‘classic experiment’ involves pre- and post- tests (measurements of outcome variables)

7
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 7 Survey experiments Systematically vary one or more elements of the survey across subjects Usually do not include ‘pre-test’ measurement –Thus, most survey experiments are not ‘classic’ in design –“Posttest-only control group design” Random assignment is critical to the design

8
Diagram of Basic Experimental Design Source: Babbie textbook

9
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 9 An inclusive definition It’s still a survey experiment even if: Sample is small Sample is not probability based Sample is not representative It’s done in a lab setting It’s only part of a pre-test for a survey project Any aspect of the survey protocol is varied Large, probability-based samples do make the survey experiment better!

10
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 10 What’s NOT a survey experiment General tinkering... –“Let’s experiment!” One shot trial of a new method Mid-stream change in method –No true randomization of cases when this happens Experiments that only use a survey to measure outcomes, pre- or post-intervention –But do not vary the survey itself

11
Is this classical experiment a survey experiment? This is a questionnaire But this is not a survey experiment Source: Babbie textbook

12
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 12 Methods vs. Substance A slippery distinction

13
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 13 Prevailing narrative... Methods experiments have been around a long time –Mostly split-ballot question wording experiments New trend is: use survey experiments for testing theories about substantive social science problems The field is moving from methods experiments to substantive experiments –And from applied to basic research...This is but a partial picture.

14
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 14 In fact... Methods experiments are burgeoning in number Methods experiments deal with far more than question wording Some methods experiments are quite complex The line between ‘methods’ and ‘substantive’ research is increasingly blurred –As theories are developed to explain variations in survey response, methods experiments are used to test these theories. –The same theories may underlie some ‘substantive’ experimentation

15
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 15 Cross-cutting categories of survey experiments MethodologicalSubstantive Applied Basic

16
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 16 Cross-cutting categories of survey experiments MethodologicalSubstantive Applied Experiments to raise response rates Basic Test of ‘leverage- salience’ theory

17
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 17 Cross-cutting categories of survey experiments MethodologicalSubstantive Applied Experiments to raise response rates Message testing Basic Test of ‘leverage- salience’ theory Test for ‘activation’ of racial hostility

18
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 18 Survey experiments about survey methods A quick scan of the landscape

24
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 24 In short... The primary corpus of accepted research in survey methods today is almost entirely based on: Survey experiments

25
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 25 Substantive survey experiments

26
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 26 Substantive survey experiments Most notable in the field of race relations –Cf. Sniderman, Gilens, Kuklinski, et al. –“mere mention” experiment –Unbalanced list technique –Activation of racial identity affecting minority responses Movement spreading to other substantive areas –but methods experiments are still more common TESS has fostered much experimentation –Over 200 experiments by 100 researchers by 2007 –Won 2007 AAPOR Warren Mitofsky Innovators Award Factorial “vignette” technique—a long tradition –(more on this later)

27
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 27 Design issues in survey experiments

28
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 28 Split-ballot vs. within-subject The vast majority of survey experiments use Split-Sample designs –“Randomized Posttest/Control Group” design –Statistical tests based on independent samples –Needs a lot of cases; most surveys have plenty Some use within-subjects designs –Greater statistical power (paired comparisons) –But later responses may be influenced by earlier questions Factorial vignette surveys often combine these –Vignettes vary across subjects –But each subject scores several vignettes

29
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 29 Statistical power issues Power of a split-sample is greatest when cases are evenly divided –If groups are equal in size, critical value = ME * –Example: N= 1200, split over two groups of 600 each. ME for each group = +/- 4 % Critical value for contrast = 4% x 1.41 = +/- 5.6% Sometimes, control group needs to be larger –To preserve comparability with earlier survey –Because experimental treatment is expensive Many experiments use more than one treatment Are pre-tests big enough for an experiment?

30
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 30 Randomization issues Full randomization between groups is crucial to the experiment’s validity Difficult to get people to carry out randomization –If possible, have the computer do it In CATI systems, don’t randomize within the interview –Pre-assign all values and store in the sample database Be sure to track which group is which! Don’t confound experimental effects with interviewer effects –Randomize across interviewers –Control for interviewer effects in analysis –Keep interviewers blind to your hypotheses

32
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 32 Factorial (vignette) surveys (with thanks to the late Steven L. Nock, my co-author)

33
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 33 Factorial surveys Substantive survey experiments about factors that affect –Judgments –Decisions –Evaluations These studies tell us: –What elements of information enter into the judgment? –How much weight does each element receive? –How closely do people agree about the above?

34
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 34 More on factorial surveys... Respondents evaluate hypothetical situations or objects, known as ‘vignettes.’ Experimental stimuli: vignettes Outcomes of interest: judgments about the vignettes Judgments will vary across the vignettes –But also across respondents

35
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 35 Why factorial surveys are cool When values of factors are allocated independently across vignettes, the factors are uncorrelated. –This simplifies modeling of the effects on judgments Factors are also independent of respondent characteristics Respondents can be given quite complex vignettes to consider –Unusual combinations presented more easily as vignettes than in the real world

36
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 36 Key design choices in factorial surveys How many factors? How many values? –More factors make the respondent’s task more difficult –More values on more factors yield larger number of possible unique vignettes –Phone surveys need simpler vignettes Example: in Nock’s study with 10 factors, and 2 to 9 values on each, there were 270,000 possible vignettes –These are sampled (by SRS) and randomly assigned across respondents

37
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 37 More design choices... Which vignettes to present? –When there are a lot of vignettes, these must be sampled –SRS across all values yields uncorrelated factors –But distribution on some factors may not be like the real world –Some randomly created vignettes are implausible The number to present to the respondent –Need to avoid fatigue, boredom, and satisficing Later judgments may be more affected by just a few factors, to which respondents become increasingly attentive –This choice depends on mode, sample, type of respondent How many assessments? –One judgment, or a series of questions about each vignette?

38
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 38 Another design choice What survey mode to use? –Paper, self-administered is possible use Mail Merge to create unique set of vignettes on each questionnaire –Phone is possible But number of vignettes and number of factors is restricted due to oral administration –Face-to-face with paper vignettes –CASI (with interviewer guidance) –Internet

39
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 39 Analysis can be complex If 500 respondents each rate 5 vignettes... –Then 2,500 vignettes are rated –Data must be converted to a vignette file of n= 2,500 –But: ratings are not independent! –Each respondent is a ‘cluster’ of interdependent ratings Solution: –Multi-level analysis –Analyze models using HLM

40
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 40 Example of a factorial survey

41
Survey of Behavioral Aspects of Sheltering and Evacuation in the National Capital Region Sponsors: Virginia Dept. of Emergency Management U.S. Dept. of Homeland Security

43
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 43 Event Scenarios Focus: dirty bomb(s) in the NCR Will residents decide to stay or to go? 3 scenarios at increasing hazard levels: Minimum, moderate, maximum Respondent is presented with only two of the three tested scenarios Over 5,000 scenario tests in the study

44
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 44 Factorial Design Four aspects (“factors”) of the scenarios were experimentally varied using random assignment PATH: Which two hazard levels are asked NOTICE: Whether the event is preceded by prior notice or threats LOCATION: The respondent’s location when the event occurs SOURCE: The source of the information about the event Notice, location, and source are kept constant for both scenarios asked

45
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 45 Three Levels of Hazard MinimumModerateMaximum 1 bomb far away1 bomb 1 mile away Multiple bombs far away + 1 bomb 1 mile away Cloud in “that area” Cloud in “your community” Wind blowing away from you Wind blowing towards you Wind blowing towards you People in “that area” should shelter – no instructions for you People in “your community” should shelter People in “your community” should shelter Respondent’s location Minimum “Far away” defined as: Maximum “Far away” defined as: In Virginia Near the University of Maryland in College Park, Maryland In DC and Maryland In DCIn Tysons Corner, VirginiaIn Virginia and Maryland In MarylandIn Tysons Corner, VirginiaIn DC and Virginia

46
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 46 Factors – two levels of NOTICE No Prior NoticePrior Notice Given No additional language added to the scenario Please imagine that three days ago in London a bomb exploded and was confirmed to be a dirty bomb, and two days ago a bomb exploded in New York and was also confirmed to be a dirty bomb…Now please imagine that yesterday [source of message] reported that a terrorist group had claimed responsibility for these bombs and said that another bomb would go off in the Washington, D.C. area. Because of that, yesterday the threat level in the Washington Metro Area was raised to the highest level.

47
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 47 Factors – two levels of LOCATION At HomeNot At Home Respondent asked to imagine being at home during the day when event occurs If the respondent is employed and primarily works indoors the location is at work (if respondent works nights they are asked to imagine being in the building during the day) If the respondent is not employed or does not primarily work indoors, the location is a building that is not home and is a location where the respondent sometimes is on weekdays

48
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 48 Factors – four levels of SOURCE Emergency manager Fire chief Local chief administrative officer Governor/mayor “The local emergency manager” “The local fire chief” “A top local official” If respondent’s location when the event occurs is in VA or MD: “The Governor” If respondent’s location when the event occurs is in DC: “The Mayor” The four factors result in 48 different possible versions of the scenario, randomly assigned.

49
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 49 Detailed Follow-up Questions Follow up questions were asked about the decision to shelter in place or evacuate, as appropriate (once only) Shelter in place detail Willingness to remain at location, reasons for leaving, what would aid staying Evacuation detail Reason for leaving, destination, mode of travel, needs, use of designated route Mandatory evacuation: everyone was asked evacuation detail eventually

50
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 50 Perception of Hazard

51
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 51 “What is your perception of the risk of death or serious injury to you or members of your household from this event?” Percent who perceive “High Risk” or “Very High Risk” (by hazard level)

52
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 52 Population Sheltering and Evacuation Behaviors Will They Stay or Will They Go?

53
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 53 “Based on this information, would you stay at HOME, would you leave immediately to go somewhere else or would you continue with your activities?” Scenario Location: Home Hazard Level MinModMax Stay at home Leave immediately Continue with activities6.6-- Something else Shelter-in-Place or Evacuation

54
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 54 “Based on this information, would you stay at WORK, would you leave immediately to go somewhere else or would you continue with your activities?” Scenario Location: Work or Other Building Hazard Level MinModMax Stay at work Go home Go to another place Continue with activities3.8-- Something else Shelter-in-Place or Evacuation (cont.)

55
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 55 Factors Affecting Behavioral Response

56
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 56 Notice of Event Location when event occurs: Home At home: In the minimum scenario, prior notice has a significant effect on the decision to stay or go

57
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 57 Notice of Event Location when event occurs: Work or Other Building At work: Prior notice has no significant effect

58
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 58 Source of Message to Shelter-in-Place (Effect on ‘leave immediately’) Compliance with shelter in place instruction is highest when the source of information is the State Governor or Mayor of DC

59
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 59 Gender of Respondent [Event occurs while at home] At home, gender effect is significant for all three scenarios. When event occurs while at work/another building, gender effect is significant in two of three scenarios.

60
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 60 Summary Findings From Scenarios: Percentage of people who would leave their home immediately is not large Many people will leave their place of work if the event is far away (‘minimal hazard’) Most of these will head to their homes The scenarios with greater ‘hazard’ did raise perception of risk But the rates of leaving are similar for moderate and maximum hazards Higher education, prior positive experience in an emergency, female gender also increase sheltering compliance Still to come: multivariate analysis using HLM

61
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 61 Closing thoughts and a look to the future

62
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 62 Are survey experiments externally valid? Survey experiments help to establish external validity –Because they are carried out on broadly representative populations But answering a survey question or judging a vignette is not necessary a ‘real world’ test –External validity is not assured by the design External validity isn’t a problem for applied survey methods experiments –The survey itself is the ‘real world’ setting for the behavior of interest

63
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 63 Survey experiments aren’t free Full-scale stand-alone survey experiments are expensive –Factorial designs are hungry for cases Adding a small experiment to an existing survey costs less But the added experiment does increase costs at every step –Design, sample creation, programming –Interviewer training, sample management –Data entry, analysis, reporting, project management Split-ballot wording experiments on existing items reduce statistical power of the original question –Asked in the control group only, smaller n

64
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 64 We need more survey experiments Most questions used in most surveys have never been subjected to rigorous testing in experiments –Substantial improvements in measurement might be achievable through more experimentation Despite small n’s and low power, testing of questions in pre-tests is potentially useful to the practitioner –Let’s do more pre-test experiments! Possibilities for substantive research are boundless

65
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 65 New technologies are changing our survey experiments Computerization has made experimentation easier in every mode (CAPI, CATI, CASI, Web) Capture of para-data sheds new light on outcomes New multimedia tools offer enhanced possibilities for presenting experimental stimuli The Internet allows experimenters to reach outside the ‘subject pool’ to the general public –But not always using probability sampling

66
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 66 The ‘knowledge gaps’ are closing... Survey experiments are increasing in number and sophistication –Survey researchers learning more about experiments Behavioral scientists moving more of their experiments to the Internet –Seeking larger, more representative samples The traditional lines between survey research and social science experiments are blurring further to the mutual benefit of both!

67
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 67 You’ve seen the movie... now read the book! Steven L. Nock and Thomas M. Guterbock “Survey Experiments.” Chapter in James Wright and Peter Marsden, eds., Handbook of Survey Research, Second Edition. Wiley Interscience, 2010.

68
Center for Survey Research University of Virginia Center for Survey Research University of Virginia 68 Survey Experiments: Past, Present, Future Thomas M. Guterbock Director, Center for Survey Research University of Virginia 2010 Annual Conference Harvard Program in Survey Research October 22, 2010