Evaluating the evidence: an expert view on improving one of EPA’s key chemical assessment programs

A collective effort has been underway the past few years to encourage the Environmental Protection Agency (EPA) to deliver more timely and credible chemical assessments under its Integrated Risk Information System (IRIS), a program that continues to face scrutiny from regulators, scientists, and industry.

A recent report from the National Research Council (NRC) acknowledged that part of this effort has paid off and that EPA has made significant progress toward improving IRIS. However, the NRC also made it clear that there is still some work to do in key areas.

An upcoming panel at the annual meeting of the Society for Risk Analysis this week will discuss some of the NRC’s recommendations for updating IRIS. The goal is to help make sure these suggested improvements don’t fall by the wayside.

One of the panelists, Dr. Martin Stephens, a Senior Research Associate at Johns Hopkins University, will discuss one of the NRC’s recommendations to strengthen the scientific foundation of IRIS by adopting a better approach to systematic review and evidence integration.

A systematic review is a high-level overview of the primary research for a particular research question that tries to identify, review, and integrate all of the evidence relevant to that question in order to answer it. The NRC has stated that incorporating systematic review methods would “substantially strengthen the IRIS process.” Dr. Stephens explains in the following Q&A why adopting this approach will indeed be helpful.

1. What did the NRC recommend?

The National Research Council (NRC) examined EPA’s IRIS program in its 2011 “Review of the Environmental Protection Agency’s Draft IRIS Assessment of Formaldehyde” and its 2014 “Review of EPA’s Integrated Risk Information System (IRIS) Process.” The NRC recommended that EPA’s IRIS program adopt a more evidence-based approach in its toxicological assessments of environmental contaminants. In particular, IRIS chemical assessments should be based on systematic reviews (SR’s) of the literature. And following these reviews, more evidence-based frameworks should be considered for integrating the evidence across data streams in order to reach more robust conclusions about causation and risk. The NRC preferred to characterize this latter process as “data integration” rather than the more vague term, “weight of evidence.”

According to the NRC, once a problem has been carefully formulated, a written protocol for how the review will be conducted should be developed. Then the evidence should be identified and evaluated within each data stream (human, animal, and mechanistic). The SR process should be standardized and the resulting report should be streamlined, with data tables substituting for extensive narrative presentation where appropriate.

The NRC provides substantial guidance on various data integration approaches without recommending a specific approach. EPA could refine its current process of guided expert judgment or move to a more structured process for “evaluating evidence and rating recommendations along the lines that the NTP has taken.” The NRC underscores the importance of quantitative approaches to evidence integration, including Bayesian approaches.

2. Why are systematic review and evidence integration important?

It is important that public health decisions be based on the best available evidence.

Such evidence should be synthesized in a transparent, objective, and consistent process, for the benefit of all stakeholders, including members of the public. Systematic reviews of evidence, with their emphasis on minimizing bias, are well-suited for this purpose, yielding robust conclusions about the current state of the evidence on a particular question. They contrast with the more familiar, expert-generated “narrative reviews,” which have their place in science but tend to be more unstructured and idiosyncratic than SR’s.

The field of toxicology is fortunate in that the basic elements of SR’s have been worked out already in the medical field, where SR’s have been championed as a core component of evidence-based medicine. Some adaptation of the SR process will be necessary as it is translated from medicine to toxicology. A particular challenge will be evidence synthesis. The “evidence base” for evidence-based medicine has come primarily from one evidence stream, namely human clinical trials. In contrast, data relevant to questions in toxicology and risk assessment often come from various evidence streams, including in vivo, in vitro, human observational, and Quantitative Structure-Activity Relationship studies. It is one thing to synthesize data within evidence streams, but it is quite a different challenge to integrate data across evidence streams. In fact, data integration is typically viewed as a separate process carried out after an SR is completed.

Informal “weighing” of such diverse evidence is unsatisfactory, especially when multiple stakeholders have different perspectives on the importance of relevant studies. A more formal process of evidence integration is needed, one that is consistent with evidence-based principles. This integration is still a work in progress for toxicology, but the National Toxicology Program’s (NTP) Office of Health Assessment and Translation (OHAT), for example, has proposed a data integration framework for its purposes. The ideal solution would be a consensus, evidence-based process that appropriately integrates all data streams relevant to toxicology and risk assessment, yet could be adapted to specific agency needs (e.g., EPA’s IRIS).

3. What has EPA done to improve under IRIS?

According to the NRC’s 2014 review, the IRIS program has made progress in implementing the NRC’s 2011 call for incorporating SR processes into its toxicological assessments. The recent review notes that inconsistencies in this implementation need to be addressed and some objectives still remain to be implemented. Yet the new approaches “bring IRIS assessments much more in line with the state of practice for systematic reviews.” The program has also done a good job at soliciting stakeholder input and enhancing the transparency of its assessments.

4. What more needs to be done?

EPA’s efforts are a work in progress. The Agency needs to continue to make the transition from narrative reviews to systematic reviews, while emphasizing transparency. And much remains to be done to devise a data integration framework that is both consistent with evidence-based principles and the IRIS program’s needs.

As it moves forward, EPA should be guided by the NRC’s 2014 report and input from all stakeholders. Moreover, the translation of evidence-based approaches from medicine to toxicology should be a collaborative effort. EPA should work actively with like-minded agencies and organizations in the U.S. and abroad to create transparent and robust processes for assembling and integrating the best evidence for making public health decisions. EPA took a step in this direction by participating in a recent meeting organized by the Evidence-based Toxicology Collaboration, which featured presentations and discussions among representatives of EPA/IRIS and other organizations seeking to advance systematic review in toxicology (see www.ebtox.com).

Martin Stephens is a director of the Evidence-based Toxicology Collaboration (EBTC), a multi-stakeholder initiative that seeks to translate evidence-based approaches from medicine and healthcare to toxicology. Current efforts of the EBTC include conducting systematic reviews of test method performance, preparing general guidance on understanding and conducting systematic reviews, and preparing guidance on assessing the methodological and reporting quality of diverse types of toxicologically relevant studies. The Johns Hopkins Center for Alternatives to Animal Testing serves as the EBTC’s secretariat. For more information on the EBTC, see www.ebtox.com.