In response, global health authorities are starting to sound a little giddy. “We believe that the world is on the verge of an efficacious Ebola vaccine,” said Marie Paule Kieny, the World Health Organization’s (WHO) assistant director-general for health systems and innovation (and an author on the study). “It could be a game changer.”

She’s right: this is wonderful news, and a great testament to human ingenuity. A genetically engineered hybrid of the benign vesicular stomatitis virus and the Zaire strain of Ebola, together called rVSV-ZEBOV, was tested in a multi-site clinical trial conducted amid a massive aid response in Guinea, one of the poorest countries in Africa. The scientific and logistical acrobatics required to pull this off boggle the mind.

Yet, for three reasons, we cannot know if the vaccine really worked, or how well. Those reasons are the lack of placebo comparison, the way the investigators diagnosed vaccine failure and the possibility of statistical flukes.

Reason #1: There was no placebo to test the vaccine against

In response to the challenging ethics of Ebola vaccine research, investigators studying rVSV-ZEBOV opted not for the standard placebo-controlled trial with which most vaccines are tested but instead for an innovative approach called “ring-vaccination.”

In the clinical trial, the investigators drew a “ring” of vaccination around people with known exposures to Ebola. Groups of people who were exposed to a person with Ebola virus disease, or to someone who had been in contact with someone else with Ebola virus disease, were invited to participate in the study. Researchers recruited 7,651 people who fit that profile and randomized them to receive immediate vaccination with rVSV-ZEBOV or vaccination that was delayed by 21 days.

Afterwards, investigators followed all study participants for several weeks to see if they developed Ebola virus disease. In the immediate vaccination groups, no one developed Ebola virus disease more than 10 days after vaccination, whereas 16 patients in the delayed vaccination group did. Because the number of cases of Ebola virus disease was so much lower (ie, zero) in the immediate vaccination group, the investigators concluded the vaccine was 100% effective.

This conclusion was reasonable, but it also could be wrong. Without a placebo comparison, it’s easy to get fooled. Perhaps people with deferred vaccination were more likely to engage in risky Ebola exposures, or those who developed Ebola virus disease in the immediate vaccination group were less likely to stay in study follow-up. The investigators did as good a job as possible to avoid these problems, but without a placebo arm in the study, we just cannot be sure.

Reason #2: The definition of vaccine failure isn’t failsafe

No cases of Ebola developed in the immediate vaccination group 10 days after vaccine was given. But nine cases of Ebola developed in the immediate vaccination group within 10 days of vaccination. Since the incubation period for Ebola virus disease is typically longer than 10 days, the investigators concluded that the nine cases of Ebola virus disease that developed within 10 days of getting the vaccine resulted from pre-vaccine exposure.

Again, this was a defensible conclusion, but it could also be wrong. Sometimes Ebola develops in less than 10 days, which means some or all of the nine people in the immediate vaccination group could have been infected with Ebola after being vaccinated. That means sometimes the vaccine may have failed to provide protection against Ebola, and we do not know exactly how often.