Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.

To obtain stakeholder input, the Human Spaceflight Public and Stakeholder Opinion Panel conducted a survey of several key stakeholder groups. The survey was conducted by NORC at the University of Chicago. This appendix describes the methods used to conduct the survey.

SAMPLING FRAME

After initial informal exploratory discussions with a variety of experts and stakeholders, the panel, in consultation with the committee, developed a list of stakeholders customized to meet the needs of this project (Table C.1). To build a sampling frame for the survey, the panel identified leadership positions within each of the stakeholder groups of interest, and then identified the individuals occupying these positions. For example, in the case of the industry group, this included CEOs and Presidents of corporations that are members of the Aerospace Industries Association, Commercial Spaceflight Federation, and the American Institute of Aeronautics and Astronautics. In the case of space scientists, the sampling frame included members of the National Academy of Sciences, Institute of Medicine, and National Academy of Engineering, with an interest in space, as well as officers and Board Directors of professional associations, such as the American Astronautical Society and American Astronomical Society.

Because the committee’s charge was to make recommendations for a sustainable program spanning the next couple of decades, the views of the younger generation were particularly important to capture. To develop a sampling frame of young space scientists, the panel assembled lists of American Institute of Aeronautics and Astronautics early career and graduate student award winners, National Science Foundation postdoctoral fellowship winners, and NASA Aeronautics Graduate Scholarship Program winners from the past 3 years.

A description of the lists used to generate the sampling frames for each of the stakeholder groups is provided in Table C.1.

The approach described yielded an overall sampling frame of over 10,000 individuals. Within each of the groups, a systematic random sample was drawn, for an overall sample of 2,054 cases. Duplicate records within each group were removed before sampling. Those individuals who appeared in the final sample for more than one group were flagged, for the purposes of the analyses, as members of each of the groups they were sampled from, but they did not receive duplicate requests to complete the survey. Because “NASA’s stakeholders” are not a clearly defined population and because we selected the stakeholders using sampling frames that were reasonable and convenient rather than comprehensive, the results from this survey cannot be generalized to all stakeholders.

Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.

Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.

In addition, we have not made any attempt to weight the data to compensate for differences in the sizes of the various groups or overlaps in their composition. As a result, combining the responses from each group does not represent the universe of all of NASA’s stakeholders. Nevertheless, the sample from each frame was a probability sample and we believe this methodology provides a broader and more diverse perspective on stakeholder views than a nonprobability sample would.

SURVEY QUESTIONNAIRE

The objective of the stakeholder survey was to provide input to the committee on stakeholder views of human spaceflight, and specifically:

The rationales traditionally provided for human spaceflight and any new or emerging rationales

The importance of human spaceflight in the context of tradeoffs and alternatives

The consequences of discontinuing NASA’s human spaceflight program

The characteristics and goals of a worthwhile and feasible program for the near future (e.g., next two decades)

The survey instrument was developed by the panel in consultation with the committee. The goal was to develop standardized questions that could be both self-administered (via a paper questionnaire or web) and administered by a trained interviewer (via phone). The final questionnaire is included in Appendix D. To minimize response order effects (respondent’s potential tendency to favor answer options appearing toward the beginning of self-administered lists and toward the end of lists administered by phone), two versions of the questionnaire were produced, reversing the order of the response items in 3, 4, 5, and 11. Respondents were randomly assigned to receive either version A or version B of the questionnaire, consistently across administration modes. (The version included in Appendix D is version A of the mail questionnaire).

Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.

To carry out the data collection, the panel selected NORC at the University of Chicago. Data collection began on September 17, 2013, and concluded on November 6, 2013.

To maximize response rates, the survey was conducted in three administration modes—mail, web and telephone—all monitored by a case management system that tracked the status and outcomes of all sampled stakeholders across modes. This ensured that sample members were not contacted once they had completed the survey. It also controlled the number of emails received by sample members who had not completed the survey so as to limit unnecessary burden.

As a first step, a paper questionnaire was sent to all sample members where a mailing address was available. The questionnaire was accompanied by a cover letter signed by the Chair of the NRC, along with a postage paid envelope for returning the completed questionnaire.

Follow-up mailings were sent to cases where the only contact information after several rounds of locating attempts was a mailing address. In addition, repeat mailings were sent to a small number of sample members who requested another mailed questionnaire when contacted by a telephone interviewer. Near the end of data collection, reminder letters were mailed via FedEx to the remaining sample members who had not yet responded within the Industry and Young Space Scientist strata to help boost participation among these groups. These letters included the link to the web survey, individual PINs, and the toll-free number that sample members could use to complete the survey by phone.

Receipt control and data entry systems were used to input responses received via mail with 100 percent data entry verification.

Following the initial mailings and emails, all non-responding sample members with a located telephone number were contacted by phone. Telephone interviewers were selected and trained from the existing group employed by NORC’s telephone center in Chicago, IL. Prior to interviewing, all interviewers passed a certification test to demonstrate that they were well-versed in the project’s purpose and survey administration. A system was also implemented to receive inbound calls via a toll-free number.

A web link to this mode of the survey was sent to all sample members with a known email address. Sample members accessed the web survey using an individualized PIN assigned at random during the compilation of the sample. A link to the web survey was also included in the initial paper mailing, through regularly scheduled email blasts, and by request.

Telephoning began on October 1, 2013. Follow-up calls were made on a regular basis to all non-completers. Exemptions were made for those who refused to participate. NORC’s telephone center supervisors monitored interviewers throughout the data collection process.

The research staff continuously monitored results for signs of any complications with the data collection process. The research staff also monitored the progress of the survey on a daily basis, tallying the number of completed surveys by mode and by strata, as well as the total number of completed surveys. Careful monitoring of the transition of cases between modes of contact was critical throughout. Any cases where it was determined that the sample member lived outside the United States, was deceased or incapacitated/unable to complete the survey were considered not eligible.

Six survey questions requested open-ended responses that required coding by the research staff. The answers were coded into categories developed by the panel. Two coders independently coded 100 cases completed early in data collection and inter-coder reliability was calculated. The resulting kappa statistics are shown in Table C.2 and indicating the coding was adequately reliable. These cases were selected proportionally from the eight stakeholder groups. NORC survey managers and the coding supervisor debriefed the panel on this initial coding process. Then, upon the completion of data collection, all open-ended responses were coded in compliance with guidelines delineated during the initial batch of open-ends.

By the end of the data collection period 1,104 individuals, or 54 percent of the initial sample, completed the survey. The AAPOR Response Rate 3, which adjusts for ineligible cases, was 55.4 percent (Table C.3). In an effort to reduce the differences in response rates among the groups, at the end of the field period an additional mailing was sent via FedEx to nonresponders in the groups with the lowest response rates. To some extent, the differences

Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.

TABLE C.2 Inter-Coder Reliability Calculated Based on the First 100 Cases Coded

Question Number

Kappa Statistic

Q1A

0.82

Q1B

0.73

Q2A

0.87

Q2B

0.75

Q6

0.73

Q18

0.88

TABLE C.3 Number of Completed Cases and Response Rates

Stratum Name

Original Sample

Number of Complete Cases

Simple Response Rate (%)

AAPOR Response Rate 3 (%)

Economic/industry

384

104

27

28.6

Space scientists and engineers

395

261

66

67.1

Young space scientists and engineers

195

90

46

49.7

Other scientists and engineers

396

201

51

51.3

Higher education

399

294

74

74.1

Security/defense/foreign policy

110

71

65

66.4

Space writers and science popularizers

99

53

54

56.4

Space advocates

96

46

48

51.7

TOTAL

2,054

1,104

54

55.4

in the final response rates reflect differences in the contact information available for the different groups, including whether efforts to reach the sample member were likely to be screened by a gatekeeper, which was especially likely in the case of the industry group.

Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.

Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.

Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.

Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.

Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.

The United States has publicly funded its human spaceflight program on a continuous basis for more than a half-century, through three wars and a half-dozen recessions, from the early Mercury and Gemini suborbital and Earth orbital missions, to the lunar landings, and thence to the first reusable winged crewed spaceplane that the United States operated for three decades. Today the United States is the major partner in a massive orbital facility - the International Space Station - that is becoming the focal point for the first tentative steps in commercial cargo and crewed orbital space flights. And yet, the long-term future of human spaceflight beyond this project is unclear. Pronouncements by multiple presidents of bold new ventures by Americans to the Moon, to Mars, and to an asteroid in its native orbit, have not been matched by the same commitment that accompanied President Kennedy's now fabled 1961 speech-namely, the substantial increase in NASA funding needed to make it happen. Are we still committed to advancing human spaceflight? What should a long-term goal be, and what does the United States need to do to achieve it?

Pathways to Exploration explores the case for advancing this endeavor, drawing on the history of rationales for human spaceflight, examining the attitudes of stakeholders and the public, and carefully assessing the technical and fiscal realities. This report recommends maintaining the long-term focus on Mars as the horizon goal for human space exploration. With this goal in mind, the report considers funding levels necessary to maintain a robust tempo of execution, current research and exploration projects and the time/resources needed to continue them, and international cooperation that could contribute to the achievement of spaceflight to Mars. According to Pathways to Exploration, a successful U.S. program would require sustained national commitment and a budget that increases by more than the rate of inflation.

In reviving a U.S. human exploration program capable of answering the enduring questions about humanity's destiny beyond our tiny blue planet, the nation will need to grapple with the attitudinal and fiscal realities of the nation today while staying true to a small but crucial set of fundamental principles for the conduct of exploration of the endless frontier. The recommendations of Pathways to Exploration provide a clear map toward a human spaceflight program that inspires students and citizens by furthering human exploration and discovery, while taking into account the long-term commitment necessary to achieve this goal.

Welcome to OpenBook!

You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.