Beyond peer review: NIST and five journals find a way to manage errors in research data

September 11, 2013

NIST chemists Michael Frenkel and Robert Chirico led a collaborative study with five journals to find ways to ensure the quality of thermophysical property data used in chemical process technologies. Credit: Burrus/NIST

Traditional peer review is not enough to ensure data quality amid the recent boom in scientific research findings, according to results of a 10-year collaboration between the National Institute of Standards and Technology (NIST) and five technical journals.

While production of research data is growing about 7 percent annually, about one-third of papers submitted to participating journals contained erroneous or incomplete chemical property data, according to a report by 32 authors from NIST and the collaborating journals. Poor data can lead to mistakes in equipment selection, over-design of industrial plant components, difficulty simulating and discovering new processes, and poor regulatory decisions, the report notes.

The traditional peer-review process is under pressure to work too fast to evaluate fully all new experimental data, the NIST-journal collaboration found. The authors' solution is a set of customized software tools and procedures for validating experimental data and eliminating errors after a paper is approved by peer review, but before a journal formally accepts the paper.

The collaboration focused on thermophysical property data used in chemical process technologies such as distillation, extraction and crystallization. Thermophysical properties include boiling and melting points, density, viscosity, solubility and many other physical values related to temperature, including those for mixtures. The study findings also may be of broad value to scientific data publishing in general.

"The solutions we offer, while centered on the field of thermodynamics, should be applicable in principle to other areas of science and engineering," says Michael Frenkel, a co-author of the new paper and leader of NIST's Thermodynamics Research Center.

Managing thermophysical property data is particularly challenging because some 100-year-old data remain useful today for engineering purposes. Efforts to establish data-reporting standards for this field began more than 50 years ago but could not succeed until recently, with the development of specialized computer hardware and software tools.

The collaboration cites a variety of factors contributing to poor data quality. Advances in measurement science have boosted data collection, but increased automation has resulted in the loss of personnel expertise and knowledge required to run manual systems. In addition, equipment manufacturers sometimes make invalid uncertainty claims. And word processing software, with functions such as cut-and-paste and "fill down" in spreadsheets, has led to many published errors. The most common problem found in papers analyzed by the collaboration was missing or underestimated uncertainties for reported data.

After several tries, the collaboration developed a rapid, cost-effective process for identifying and eliminating errors. NIST developed a new online tool (NIST ThermoLit) that allows researchers to generate a literature report containing relevant references retrieved from a NIST database. Researchers can combine this capability with an older experiment planning system (NIST ThermoPlan) at both the conceptualization and publication stages of their work. If the submitted paper passes a journal's peer review, NIST generates a report noting any inconsistencies between the new experimental data and critically evaluated data based on past research. Data are extracted from the submitted paper and validated by NIST's expert software system for data evaluation (NIST ThermoData Engine).

Related Stories

The National Institute of Standards and Technology has released an expanded and upgraded version of a popular database, a computer package for calculating the properties and modeling the behavior of fluids.

A new guide for Web developers recently released by the National Institute of Standards and Technology (NIST) will make it easier for electric utilities and vendors to give customers convenient, electronic access to their ...

A fundamental advance in measurement capabilities that could save semiconductor manufacturers billions of dollars annually has earned a 2013 R&D 100 Award for its National Institute of Standards and Technology (NIST) inventors.

The National Institute of Standards and Technology (NIST) has updated two of its series of computer security guides to help computer system managers protect their systems from hackers and malware. Vulnerabilities in software ...

Played out on a computer over hundreds of generations, a survival-of-the-fittest programming method adapted by National Institute of Standards and Technology (NIST) researchers has spawned, of all things, the design for a ...

Recommended for you

It sounds like a science-fiction nightmare. But "killer robots" have the likes of British scientist Stephen Hawking and Apple co-founder Steve Wozniak fretting, and warning they could fuel ethnic cleansing and an arms race.

Photos. Messages. Bank account codes. And so much more—sit on a person's mobile device, and the question is, how to secure them without having to depend on lengthy password codes of letters and numbers. Vendors promoting ...

A startup team calls their work a product. They also call it a social movement. Many people in the over-7,000 islands in the Philippines lack access to electricity .The startup would like to make a difference. Their main ...