Reanalysis changes findings in respiratory protection study

Editor's note: This story was revised Nov 6 to include additional information concerning the recent Institute of Medicine report on respiratory protection for healthcare workers against H1N1 influenza.

Nov 5, 2009 (CIDRAP News) – A reanalysis prompted by reviewers has changed the conclusions of a study comparing N-95 respirators with surgical masks, raising questions about earlier findings that the N-95 devices were clearly more effective in protecting healthcare workers from respiratory illness.

Raina MacIntyre and colleagues first presented their findings at a medical conference in mid-September, reporting that N-95s, compared with no respiratory protection, reduced the risk of confirmed influenza in hospital workers by 75%, whereas surgical masks had no protective effect. The findings were hailed by some experts as a landmark in a field where few clinical studies have been done.

But at the Infectious Diseases Society of America (IDSA) annual meeting last week, MacIntyre's team presented a new analysis of their data, made at the request of peer reviewers. The reanalysis excluded the control group of unprotected workers. The result was that workers who wore N-95s still appeared to be better off than those with surgical masks, but the differences were no longer statistically significant.

Some press reports and critics of the study—which has not yet been published in a journal—characterized the reanalysis as a retraction of the earlier report. Critics also suggest that the study influenced the recent recommendation by the Institute of Medicine (IOM) that healthcare workers should wear N-95s when caring for H1N1 patients. MacIntyre was a member of the committee that wrote the IOM report, issued in September.

But MacIntyre says the reanalysis was not a retraction. She asserts that the study results still indicate a real difference in levels of protection, but the changes requested by the peer reviewers left the study "underpowered" to show significance. Further, she says the IOM panel considered only published studies in making its recommendations.

The new development adds another chapter to the controversy over whether healthcare workers should, as recommended by the Centers for Disease Control and Prevention (CDC), wear N-95s when caring directly for H1N1 influenza patients. The devices are designed to filter out at least 95% of tiny airborne particles.

N-95s are often in short supply and are more expensive than surgical masks. Some professional groups, including the Society for Healthcare Epidemiology of America (SHEA), oppose routine use of N-95s in flu settings because research on their clinical effectiveness has been inconclusive, and many workers find them uncomfortable to wear for long periods.

The CDC, in issuing interim recommendations for protecting health workers from H1N1 flu last month, maintained its earlier guidance that workers in close contact with flu patients should wear an N-95. But the agency listed that step as just part of a comprehensive set of precautions, and said that when N-95s are in short supply, they should be reserved for high-risk situations such as aerosol-generating procedures.

Original analysisAs reported previously, MacIntyre and colleagues, of the University of New South Wales in Australia, recruited more than 1,900 workers at 24 hospitals in Beijing. They were divided into four groups: surgical masks, fit-tested N-95s, non-fit-tested N-95s, or control (no respiratory protection). The volunteers wore their assigned form of protection for 4 weeks and were monitored for illness for 5 weeks.

According to the report presented at the Interscience Conference on Antimicrobials and Chemotherapy (ICAAC) in September, the authors found that surgical masks had no protective effect. In contrast, the N95s, compared with the controls, were linked with a 60% reduction in risk for any respiratory illness, a 75% reduction in flu-like illness, a 56% decrease in lab-confirmed respiratory illness, and a 75% reduction in confirmed flu. The reductions met the test of statistical significance.

However, the researchers found that fit-testing of the N-95s—recommended by manufacturers to keep air from leaking around them—made no difference in protection.

Before presenting their findings at the recent IDSA meeting, the research team ran a new analysis to satisfy the concerns of reviewers, MacIntyre told CIDRAP News by e-mail.

One change was to drop the control group of unprotected hospital workers, because of the reviewers' objection that those workers were not randomly assigned to not wear protection. MacIntyre explained that her team had to use a convenience sample of workers from hospitals where mask-wearing was not the norm, because the Chinese workers would not accept being randomly assigned to doing without respiratory protection.

The other change was to make statistical adjustments to allow for the possibility of different risks of infection among the various hospitals in the study, MacIntyre said.

After those changes, the differences between the N-95s and surgical masks lost their statistical significance. "When the P value [test of significance] was adjusted for clustering and multiple testing, [there was] no significant difference between N-95 respirators and medical masks for the four primary outcomes," states the team's slide presentation, supplied by the IDSA.

Though the differences were no longer significant, the attack rates in the N-95 group were consistently lower than in the medical mask group, the team reported. They said their post hoc analysis indicated that the probability that this happened by chance is only 3.4%.

Conclusions undermined?But Dr. Neil Fishman, president-elect of SHEA, said the team's new report signals a real undermining of the conclusions presented earlier. He is director of healthcare epidemiology and infection prevention and control in the University of Pennsylvania Health System.

"I'm certain that if you talked to the investigators, they'd say it's not a retraction, it's just a reanalysis. But at some point you have to look at what your data is telling you," he told CIDRAP News. "You can't keep changing the analysis if you don't get the answer you want. This is definitely a different answer than the first analysis."

He asserted that a failure to show that N-95s provide better protection than surgical masks is "consistent with what we know about clinical transmission of flu among humans. . . . There's no clinical data in humans that this disease is transmitted by any other means than large droplet particles"—the kind that simple masks could block as effectively as N-95s.

As for the special statistical analysis indicating that the lower attack rates in the N-95 wearers probably represented a real difference, Fishman said, "I'm concerned that it's a statistical aberration."

Fishman also said, "It appears that the IOM recommendations were based largely on this study," as MacIntyre served on the committee that wrote the IOM report.

He said the IOM report heavily influenced the CDC's recent interim recommendations, and added, "So now we have a public health policy that's based on a flawed study, and more significantly, OSHA [the Occupational Safety and Health Administration] has said they're going to enforce the guidance."

MacIntyre, however, insisted that the reanalysis still suggests a real difference between the two types of respiratory protection.

"The study [as presented at IDSA] was not a retraction at all—it was unfortunate that I was unable to attend the conference and clarify this," she said. The findings were presented at IDSA by a colleague.

"What was presented at IDSA is that the rates of infection were double in the surgical [mask] arm compared to the N95 arm for all outcomes, and statistical testing (bootstrapping) shows this is over 97% likely to be a real difference, but the P value (test for significance) just loses significance," MacIntyre said. "What this means is that the study still shows a likely superiority of N95s . . . but the study was probably underpowered to pick up statistical significance when we removed the control group."

She added that the rates of all illness outcomes in the control group "were higher than in the surgical [mask] or N-95 groups, so removing over 500 health workers from the analysis also contributed to the P values losing significance."

MacIntyre further commented, "The changes we made in the analysis were in response to reviewer comments when preparing the paper, but there is no clear 'right' or 'wrong' analysis. My conclusions are still that N-95s are superior."

As for the IOM report, she stated, "The IOM recommendations were never based on my study, as it was not published, but on the large body of experimental data on fit and filtration which show overwhelming superiority of N-95 respirators. . . . I would certainly wear an N-95 respirator if I were exposed to infectious patients."

IOM committee member commentsDr. Bonnie Rogers of the University of North Carolina at Chapel Hill, vice-chair of the IOM committee, agreed that the committee did not rely on the MacIntyre study—or another recent clinical comparison study that had not yet been published—in reaching its conclusions.

Rogers pointed out that the report refers to the MacIntyre study and to a study by Mark Loeb and colleagues, which found little or no difference in the clinical effectiveness of N-95s and surgical masks. That study was published in the Journal of the American Medical Association in October, several weeks after the IOM report was released.

As noted by Rogers, the IOM report states, "Without having the full details of the studies the committee could not draw conclusions from either study. Clinical effectiveness data are thus quite limited and conflicting at this time." Given the limited and conflicting clinical evidence, the committee based its recommendations on the experimental evidence of airborne transmission of flu viruses and the fit characteristics of N-95s compared with medical masks, the report adds.

Rogers commented, "I find that our recommendations are solidly based on the scientific evidence, particularly that related to filtration and fit. This was entirely exclusive of the Loeb and MacIntrye abstracts as is stated in the Letter Report, as the science for these studies was simply not available for the committee to review and debate."

MacIntyre said her team is currently working to get their study published in a journal.