Scientific research versus The Forum Report

In March of 2016, The Forum on Education Abroad, which is the standards setting body for the study abroad industry, issued the following paper, Insurance Claims Data and Mortality Rate for College Students Studying Abroad.

When Ros and I read The Forum Report, to say we were disappointed does not begin to capture our feelings. For several years, we’d been asking both the study abroad industry and state and federal legislators for transparent and comprehensive safety data. This was not the product we’d ever imagined. For one thing, there were so many issues with the paper’s research methods*. But even more, The Forum Report began with its conclusion, and Ros and I had a bad feeling about what purpose this conclusion might serve.

Key Finding:College students studying abroad are less likely to die than college students studying on campuses in the U.S.

The Forum Report’s “key finding” is based on three data sources, all of them flawed, though in different ways:

The Critical Incident Database (CID):

CID is The Forum’s own tool

CID was launched in 2012, with 2014 as its first reporting year

CID asks broad categories of questions (this is a good thing)

CID participation is voluntary (this results in something called self selection bias, which is problematic)

These four insurance company deaths formed the basis for The Forum Report’s key finding. Using Open Doors*** data to estimate that same year’s number of students studying abroad, they calculated an all-cause mortality rate for study abroad of 13.5. The Forum then compared this number with Turner’s home campus death rate of 22.4. Thus, their claim of study abroad as safer.

Aside from fundamental pitfalls born of comparing data that is “apples and oranges” (as CID, Turner, and the insurance data are), The Forum Report fails as a scientific study for the following reasons:

No named author(s)

No abstract

No hypothesis

No review of the literature

No definition of terms

For example, how is “death” defined? Who is included? Who is excluded?

The definition of death can explain what we believe to be an undercount

A poorly written methods section****

No expert reader we consulted can precisely say what was done

For this reason, experts told us no scientist could replicate this study, which means no possibility of peer review*****

As listed above, there are problems with all three data sources

Voluntary, with low participation rates

Confidentiality for all

Data represents a limited scope of time

Was 2014 a representative year?

The authors did not question 13.5 as a reasonable all-cause mortality rate

Experts familiar with youth mortality research suggest this number is “implausibly low”

A scientific research paper places findings and opinions in the conclusion section

Potential legislation is cited by The Forum as the reason for this paper, as well as “reassurances” to the industry and future study abroad families

The Forum Report is published in a Forum publication, which again means no peer review

For these reasons, ProtectStudentsAbroad has concluded, though The Forum Report looks sort of like a research study, it is not. It is an “industry report”, written by the industry for the industry.

Industry reports are common to for-profit corporations, where business success depends upon budgets, quotas, and organizational control derived by powering through narratives. Industry reports are not intended to protect human safety. They are meant to drive business.

Indeed, four months after The Forum released its report, a University of Wisconsin student died. Beau Solomon was murdered while studying abroad with third party program John Cabot University in Rome.

Aside from the flagrant insensitivity of such statements, The Forum Report’s underlying question of campus safety versus study abroad safety isn’t a line of inquiry bereaved families have ever pursued. Risks faced by students on campus versus students during study abroad are different; we get that. It is disappointing to us that The Forum Report did nothing to identify specific risks to study abroad students.

And so, in the aftermath of The Forum Report, using student deaths and injuries available via Internet, we are reviewing this information in an increasingly systematic way. While we do not claim PSA’s data as quantitative, or even scientific, we believe it is qualitative and notable—an ad hoc dataset compiled using available resources, including the web, volunteer experts, and the labor and knowledge of bereaved families. From our review, we would like to share a few observations.

While The Forum Report based their key finding off four student deaths, during that same calendar year we’ve identified fourteen. The following are categories of students we suspect were not included in The Forum Report:

High school students

Graduate students

Students who are foreign-born, but studying in the US, including US study abroad

Students studying abroad and not receiving credit, including internships, mission trips, religious trips, and performance trips

Students whose death occurred during “free-time”, including pre-, intra-, and post-program travel

Students whose programs were not insured by the two companies who participated in The Forum Report

Students attending non-Forum schools

Among our dead, injury appears to lead causes of death, specifically motor vehicle accidents, drowning and falls. Furthermore, heightened risk appears to be linked to specific demographics. For example, gender appears to be linked with manner of death. Also, certain countries of study appear to be linked with manner of death. In light of these observations, ProtectStudentsAbroad issues two recommendations.

To the study abroad industry: If fatality prevention is your profession’s premier goal, then embrace principles of transparency. To do less goes against American higher education’s historic ideals and operates as a betrayal of trust for students and their families.

To students and parents: The American publics’ lack of scientific literacy has allowed all sorts of industries, including the study abroad industry, to market sloppy science as “truth”. Until such time that the study abroad industry steps up and into the 21st century of evidence-based education******, we implore you to vigorously interrogate all claims made by programs. Ask difficult questions. Expect real answers.

The stakes are high. This work is hard. The cause is good.

~

*Research methods refers to the means by which data is gathered, and then statistically treated and analyzed.

**All-cause mortality is a measure used in research as an indicator of the safety or hazard of an intervention. For deaths, it is customary to use rates per 100,000, over a one-year period. In Turner’s study, he was measuring all deaths that occurred in his specified population, regardless the cause.

***Open Doors is a “comprehensive information resource on international students and scholars studying or teaching at higher education institutions in the U.S., and U.S. students studying abroad for academic credit at their home colleges or universities”. Under the umbrella of the Institute of International Education (IIE) and supported by a grant from the U.S. State Department, institutions of higher education voluntarily complete yearly surveys, from which an annual Open Doors publication is produced. Open Doors express purpose is to promote and grow international education; it does not collect safety data.

****The methods section of a research paper follows an exhaustive review of existing research on the topic. The methods section describes procedures used to collect data, and also details statistical treatment of that data. Methods are the nuts and bolts of a research project. When scientific methods are sound, the findings can illuminate, without bias, otherwise indecipherable connections between outcomes and a whole host of variables.

*****The peer review process begins with a research paper being submitted to editors of scholarly journals. If interested, editors refer the paper to experts in the field. Using discreet criteria, experts read and comment. Peer reviewers pay particular attention to the methods and results section, since these are most complicated and technically difficult. Methods sections should be written clearly enough that another researcher could pick up the paper and replicate the project. When well done, the peer review process serves to screen out poor research, screen in good, and in this way promote “best practices” for the entire field.

******Evidence-based education considers all aspects of education, from policy-making to classroom implementation, and applies evidence obtained from scientific research. Such research is not meant as a substitution for professional experience. Rather, it is a critical adjunct, meant to deepen and broaden fact-based knowledge.