Bibliometrics

Grantee Research Project Results

Bibliometric Analysis for the U.S. Environmental Protection Agency/Office of Research and Development’s Particulate Matter Research

This is a bibliometric analysis of the papers prepared by U.S. Environmental Protection Agency (EPA) intramural and extramural researchers on particulate matter (PM) research, which is a component of EPA’s Air Research Program. For this analysis, 1,561 papers were reviewed, and they were published from 1998 to 2007. These publications were cited 27,449 times in the journals covered by Thomson Scientific’s Web of Science1 and Elsevier’s Scopus2. Of these 1,561 publications, 1,369 (87.7%) have been cited at least once in a journal.

Searches of Web of Science and Scopus were conducted to obtain times cited data for the PM journal publications. The analysis was completed using Thomson’s Essential Science Indicators(ESI) and Journal Citation Reports (JCR) as benchmarks. ESI provides access to a unique and comprehensive compilation of essential science performance statistics and science trends data derived from Thomson’s databases. For this analysis, the ESI highly cited papers thresholds as well as the hot papers thresholds were used to assess the influence and impact of the PM papers. JCR is a recognized authority for evaluating journals. It presents quantifiable statistical data that provide a systematic, objective way to evaluate the world’s leading journals and their impact and influence in the global research community. The two key measures used in this analysis to assess the journals in which the EPA PM papers are published are the Impact Factor and Immediacy Index. The Impact Factor is a measure of the frequency with which the “average article” in a journal has been cited in a particular year. The Impact Factor helps evaluate a journal’s relative importance, especially when compared to other journals in the same field. The Immediacy Index is a measure of how quickly the “average article” in a journal is cited. This index indicates how often articles published in a journal are cited within the same year and it is useful in comparing how quickly journals are cited.

The report includes a summary of the results of the bibliometric analysis, an analysis of the 1,561 PM research papers analyzed by ESI field (e.g., Clinical Medicine, Environment/ Ecology, and Geosciences), an analysis of the journals in which the PM papers were published, a table of the highly cited researchers publishing on PM research, and a list of patents that have resulted from the program.

Summary of Results

More than one-third of the PM publications are highly cited papers. 578 (37.0%) of the PM papers qualify as highly cited when using the ESI criteria for the top 10% of highly cited publications. This is 3.7 times the 10% of papers expected to be highly cited. 96 (6.2%) of the PM papers qualify as highly cited when using the ESI criteria for the top 1%, which is 6.1 times the number expected. 14 (0.9%) of these papers qualify as very highly cited when using the criteria for the top 0.1%, which is 9 times the number anticipated. None of the papers actually meets the 0.01% threshold for the most highly cited papers, which is not surprising given that the expected number for this program is 0.2 papers.

The PM papers are more highly cited than the average paper. Using the ESI average citation rates for papers published by field as the benchmark, in 13 of the 18 fields in which the 1,561 EPA PM papers were published, the ratio of actual to expected cites is greater than 1, indicating that the PM papers are more highly cited than the average papers in those fields. For all 18 fields combined, the ratio of total number of cites to the total number of expected cites (27,449 to 10,856.34) is 2.5, indicating that the PM papers are more highly cited than the average paper.

More than one-third of the PM papers are published in high impact journals. 537 of the 1,561 papers were published in the top 10% of journals ranked by JCR Impact Factor, representing 34.4% of EPA’s PM papers. This number is 3.4 times higher than the expected 156 papers. Nearly one-half of the papers are published in high impact journals as determined by JCR Immediacy Index. 762 of the 1,561 papers appear in the top 10% of journals ranked by JCR Immediacy Index, representing 48.8% of EPA’s PM papers. This number is 4.9 times higher than the expected 156 papers.

Forty-five of the PM papers qualify as hot papers. Using the hot paper thresholds established by ESI as a benchmark, 45 hot papers, representing 2.9% of the PM papers, were identified in the analysis. Hot papers are papers that were highly cited shortly after they were published. The number of PM hot papers identified is 29 times higher than the expected 2 hot papers.

The authors of the PM papers cite themselves much less than the average author. 1,227of the 27,449 cites are author self-cites. This 4.5% author self-citation rate is well below the accepted range of 10-30% author self-citation rate.

Forty (1.5%) of the 2,710 authors of the PM papers are included in ISIHighlyCited.com,which is a database of the world’s most influential researchers who have made key contributions to science and technology during the period from 1981 to 1999.

There were 6 patents issued to investigators from 1998 to 2007 for research that was conducted under EPA’s PM research. Two of these patents were cited by a total of 9 other patents.

Highly Cited PM Publications

All of the journals covered by ESI are assigned a field, and to compensate for varying citation rates across scientific fields, different thresholds are applied to each field. Thresholds are set to select highly cited papers to be listed in ESI. Different thresholds are set for both field and year of publication. Setting different thresholds for each year allows comparable representation for older and younger papers for each field.

The 1,561 PM research papers reviewed for this analysis were published in journals that were assigned to 18 of the 22 ESI fields. The distribution of the papers among these 18 fields and the number of citations by field are presented in Table 1.

Table 1. PM Papers by ESI Fields

ESI Field

No. of Citations

No. of PM Papers

Average Cites/Paper

Biology & Biochemistry

486

34

14.3

Chemistry

1,056

78

13.5

Clinical Medicine

6,346

243

26.1

Computer Science

6

2

3.0

Economics & Business

25

3

8.3

Engineering

3,628

272

13.3

Environment/Ecology

6,507

333

19.5

Geosciences

5,527

350

15.8

Immunology

372

13

28.6

Materials Science

1

1

1.0

Mathematics

31

6

5.2

Molecular Biology & Genetics

25

3

8.3

Multidisciplinary

389

9

43.2

Neuroscience & Behavior

185

11

16.8

Pharmacology & Toxicology

2,642

179

14.8

Physics

164

11

14.9

Plant & Animal Science

32

5

6.4

Social Sciences, general

27

8

3.4

Total = 27,449

Total = 1,561

17.6

There are 578 (37.0% of the papers analyzed) highly cited EPA PM papers in 13 of the 18 fields—Biology & Biochemistry, Chemistry, Clinical Medicine, Economics & Business, Engineering, Environment/Ecology, Geosciences, Immunology, Mathematics, Multidisciplinary, Pharmacology & Toxicology, Physics, and Social Sciences—when using the ESI criteria for the top 10% of papers. Table 2 shows the number of EPA PM papers in those 13 fields that meet the top 10% threshold in ESI. Ninety-six (6.2%) of the papers analyzed qualify as highly cited when using the ESI criteria for the top 1% of papers. These papers cover 8 fields—Chemistry, Clinical Medicine, Economics & Business, Engineering, Environment/ Ecology, Geosciences, Multidisciplinary, and Pharmacology & Toxicology. Table 3 shows the 96 papers by field that meet the top 1% threshold in ESI. The citations for these 96 papers are provided in Tables 4 through 11. Table 12 shows the 14 (0.9%) papers by field that meet the top 0.1% threshold in ESI. These 14 very highly cited PM papers in the fields of Chemistry, Clinical Medicine, Economics & Business, Engineering, Environment/Ecology, and Geosciences are listed in Table 13. None of the PM papers meet the top 0.01% threshold in ESI, which is not surprising because the expected number of papers that should meet this threshold for this analysis is 0.2. The highly cited papers in Tables 4 through 11 are presented in order of year of publication with the oldest papers appearing first. Within the year of publication, the papers are ordered by increasing number of times cited.

Table 2. Number of Highly Cited PM Papers by Field (top 10%)

ESI Field

No. of Citations

No. of Papers

Average Cites/Paper

% of Papers in Field

Biology & Biochemistry

179

5

35.8

14.7%

Chemistry

663

21

31.6

26.9%

Clinical Medicine

4,954

90

55.0

37.0%

Economics & Business

7

1

7.0

33.3%

Engineering

3,159

123

25.7

45.2%

Environment/Ecology

5,070

152

33.4

45.6%

Geosciences

3,871

126

30.7

36.0%

Immunology

303

5

60.6

38.5%

Mathematics

25

2

12.5

33.3%

Multidisciplinary

366

5

73.2

62.5%

Pharmacology & Toxicology

1,588

44

36.4

24.6%

Physics

117

3

39.0

27.3%

Social Sciences, general

6

1

6.0

12.5%

Total = 20,308

Total = 578

35.1

37.0%

Table 3. Number of Highly Cited PM Papers by Field (top 1%)

ESI Field

No. of Citations

No. of Papers

Average Cites/Paper

% of PM Papers in Field

Chemistry

62

2

31.0

2.6%

Clinical Medicine

1,513

8

189.1

3.3%

Economics & Business

7

1

7.0

33.3%

Engineering

1,746

31

56.3

11.4%

Environment/Ecology

1,549

31

50.0

9.3%

Geosciences

1,416

19

74.5

5.4%

Multidisciplinary

272

2

136.0

22.2%

Pharmacology & Toxicology

259

2

129.5

1.1%

Total = 6,824

Total = 96

71.1

6.2%

Table 4. Highly Cited PM Papers in the Field of Chemistry (top 1%)

No. of Cites

ESI Threshold

First Author

Paper

59

43

Gao S

Low-molecular-weight and oligomeric components in secondary organic aerosol from the ozonolysis of cycloalkenes and alpha-pinene. Journal of Physical Chemistry A 2004;108(46):10147-10164.

3

2

Rudich Y

Aging of organic aerosol: bridging the gap between laboratory and field studies. Annual Review of Physical Chemistry 2007;58:321-352.

The expected citation rate is the average number of cites that a paper published in the same journal in the same year and of the same document type (article, review, editorial, etc.) has received from the year of publication to the present. Using the ESI average citation rates for papers published by field as the benchmark, in 13 of the 18 fields in which the EPA PM papers were published, the ratio of actual to expected cites is greater than 1, indicating that the PM papers are more highly cited than the average papers in those fields (see Table 14). For one field, the ratio is equal to 1, indicating that the papers in that ESI field are cited the same as the average paper. For all 18 fields combined, the ratio of total number of cites to the total number of expected cites (27,449 to 10,856.34) is 2.5, indicating that the PM papers are more highly cited than the average paper.

Table 14. Ratio of Actual Cites to Expected Cites for PM Papers by Field

ESI Field

Total Cites

Expected Cite Rate

Ratio

Biology & Biochemistry

486

497.14

1.0

Chemistry

1,056

642.11

1.6

Clinical Medicine

6,346

2,274.10

2.8

Computer Science

6

5.14

1.2

Economics & Business

25

7.29

3.4

Engineering

3,628

815.40

4.4

Environment/Ecology

6,507

2,250.50

2.9

Geosciences

5,527

2,091.43

2.6

Immunology

372

213.76

1.7

Materials Science

1

3.20

0.3

Mathematics

31

11.07

2.8

Molecular Biology & Genetics

25

82.44

0.3

Multidisciplinary

389

39.00

10.0

Neuroscience & Behavior

185

220.51

0.8

Pharmacology & Toxicology

2,642

1,542.98

1.7

Physics

164

100.75

1.6

Plant & Animal Science

32

38.05

0.8

Social Sciences, general

27

21.47

1.2

TOTAL

27,449

10,856.34

2.5

JCR Benchmarks

Impact Factor. The JCR Impact Factor is a well known metric in citation analysis. It is a measure of the frequency with which the “average article” in a journal has been cited in a particular year. The Impact Factor helps evaluate a journal’s relative importance, especially when compared to others in the same field. The Impact Factor is calculated by dividing the number of citations in the current year to articles published in the 2 previous years by the total number of articles published in the 2 previous years.

Table 15 indicates the number of PM papers published in the top 10% of journals, based on the JCR Impact Factor. Five hundred thirty-seven (537) of 1,561 papers were published in the top 10% of journals, representing 34.4% of EPA’s PM papers. This indicates that more than one-third of the PM papers are published in the highest quality journals as determined by the JCR Impact Factor, which is 3.4 times higher than the expected percentage.

Table 15. PM Papers in Top 10% of Journals by JCR Impact Factor

EPA PM Papers in that Journal

Journal

Impact Factor(IF)

JCR IF Rank

2

New England Journal of Medicine

51.296

2

6

Science

30.028

9

3

Lancet

25.800

18

4

JAMA—Journal of the American Medical Association

23.175

23

1

Journal of Clinical Investigation

15.754

42

1

Annual Review of Physical Chemistry

11.250

83

10

Circulation

10.940

88

1

Nano Letters

9.960

110

2

Proceedings of the National Academy of Sciences of the United States of America

9.643

116

27

American Journal of Respiratory and Critical Care Medicine

9.091

131

6

Journal of Allergy and Clinical Immunology

8.829

136

1

Advanced Drug Delivery Reviews

7.977

156

2

Cancer Research

7.656

172

1

Journal of Neuroscience

7.453

177

1

FASEB Journal

6.721

206

1

Critical Care Medicine

6.599

211

5

Journal of Immunology

6.293

223

5

Thorax

6.064

237

1

American Journal of Pathology

5.917

249

117

Environmental Health Perspectives

5.861

255

4

Journal of Biological Chemistry

5.808

260

12

Analytical Chemistry

5.646

276

5

Free Radical Biology & Medicine

5.440

289

1

Stroke

5.391

293

12

American Journal of Epidemiology

5.241

308

1

Journal of Thrombosis and Haemostasis

5.138

325

4

European Respiratory Journal

5.076

335

1

TrAC - Trends in Analytical Chemistry

5.068

337

1

Cellular Signalling

4.887

363

1

Faraday Discussions

4.731

393

25

Toxicology and Applied Pharmacology

4.722

397

16

American Journal of Respiratory Cell and Molecular Biology

4.593

412

1

Journal of Leukocyte Biology

4.572

415

1

Journal of Catalysis

4.533

418

1

International Journal of Epidemiology

4.517

424

1

Antioxidants & Redox Signaling

4.491

431

2

Atmospheric Chemistry and Physics

4.362

449

23

Epidemiology

4.339

452

2

American Journal of Physiology - Cell Physiology

4.334

455

1

Progress in Energy and Combustion Science

4.333

456

34

American Journal of Physiology - Lung Cellular and Molecular Physiology

4.250

472

2

Journal of Physical Chemistry B

4.115

501

120

Environmental Science & Technology

4.040

518

2

Applied Catalysis B: Environmental

3.942

548

3

Chest

3.924

552

1

Experimental Cell Research

3.777

596

1

Human Reproduction

3.769

599

1

Bulletin of the American Meteorological Society

3.728

614

1

American Journal of Physiology - Heart and Circulatory Physiology

3.724

616

1

American Journal of Public Health

3.698

626

1

Journal of Cellular Physiology

3.638

646

2

Clinical Immunology

3.606

659

31

Toxicological Sciences

3.598

662

1

Journal of Chromatography A

3.554

678

3

Journal of Neuroscience Research

3.476

704

11

Journal of Applied Physiology

3.178

807

1

Journal of Chemical Physics

3.166

814

3

Chemical Research in Toxicology

3.162

818

1

Remote Sensing of Environment

3.064

855

4

Journal of Physical Chemistry A

3.047

863

1

American Journal of Cardiology

3.015

876

Total = 537

Immediacy Index. The JCR Immediacy Index is a measure of how quickly the average article in a journal is cited. It indicates how often articles published in a journal are cited within the year they are published. The Immediacy Index is calculated by dividing the number of citations to articles published in a given year by the number of articles published in that year.

Table 16 indicates the number of PM papers published in the top 10% of journals, based on the JCR Immediacy Index. Seven hundred sixty-two (762) of the 1,561 papers appear in the top 10% of journals, representing 48.8% of the PM papers. This indicates that nearly one-half of the PM papers are published in the highest quality journals as determined by the JCR Immediacy Index, which is 4.9 times higher than the expected percentage.

Table 16. PM Papers in Top 10% of Journals by JCR Immediacy Index

EPA PM Papers in that Journal

Journal

Immediacy Index(II)

JCR II Rank

2

New England Journal of Medicine

12.743

2

4

JAMA - Journal of the American Medical Association

7.781

4

3

Lancet

7.419

6

6

Science

5.555

16

1

Journal of Clinical Investigation

3.911

29

1

Faraday Discussions

2.766

59

10

Circulation

2.674

63

1

International Journal of Epidemiology

2.200

84

27

American Journal of Respiratory and Critical Care Medicine

2.006

98

6

Journal of Allergy and Clinical Immunology

1.790

118

1

Annual Review of Physical Chemistry

1.762

124

2

Proceedings of the National Academy of Sciences of the United States of America

1.758

126

1

Critical Care Medicine

1.641

146

4

Philosophical Transactions of the Royal Society of London Series A: Mathematical and Physical Sciences

1.534

166

1

Nano Letters

1.485

177

5

Thorax

1.460

184

23

Epidemiology

1.437

187

1

Journal of Thrombosis and Haemostasis

1.397

194

1

Journal of Neuroscience

1.319

216

1

Stroke

1.242

237

1

FASEB Journal

1.241

238

2

Cancer Research

1.220

246

1

Cellular Signalling

1.213

249

1

Antioxidants & Redox Signaling

1.131

281

4

Journal of Biological Chemistry

1.110

291

3

Chest

1.110

291

4

European Respiratory Journal

1.108

294

12

American Journal of Epidemiology

1.091

306

11

Journal of Applied Physiology

1.026

343

2

Atmospheric Chemistry and Physics

1.015

350

117

Environmental Health Perspectives

0.994

373

1

Environmental Science and Pollution Research

0.982

376

16

American Journal of Respiratory Cell and Molecular Biology

0.925

404

2

American Journal of Physiology - Cell Physiology

0.906

417

5

Journal of Immunology

0.886

435

1

Journal of Cellular Physiology

0.867

453

1

Physical Chemistry Chemical Physics

0.866

454

1

American Journal of Pathology

0.833

487

34

American Journal of Physiology - Lung Cellular and Molecular Physiology

0.832

493

12

Analytical Chemistry

0.795

524

1

American Journal of Physiology - Heart and Circulatory Physiology

0.777

547

1

TrAC - Trends in Analytical Chemistry

0.752

578

5

Free Radical Biology & Medicine

0.751

580

1

Journal of Catalysis

0.751

580

1

American Journal of Public Health

0.740

588

1

Human Reproduction

0.734

597

31

Toxicological Sciences

0.734

597

4

Journal of Physical Chemistry A

0.730

602

1

Journal of Chemical Physics

0.721

616

103

Journal of Geophysical Research

0.684

673

1

Agricultural and Forest Meteorology

0.669

690

1

Journal of Leukocyte Biology

0.668

691

3

Chemical Research in Toxicology

0.663

703

120

Environmental Science & Technology

0.646

729

1

Bulletin of the American Meteorological Society

0.646

729

1

Journal of Environmental Pathology, Toxicology and Oncology

0.639

742

2

Journal of Physical Chemistry B

0.637

746

4

Boundary-Layer Meteorology

0.629

758

1

American Journal of Cardiology

0.615

781

1

Equine Veterinary Journal

0.611

790

2

Clinical Immunology

0.604

804

6

Journal of Exposure Science and Environmental Epidemiology

0.596

821

6

Environmental Research

0.583

844

132

Aerosol Science and Technology

0.571

872

Total = 762

Hot Papers

ESI establishes citation thresholds for hot papers, which are selected from the highly cited papers in different fields, but the time frame for citing and cited papers is much shorter—papers must be cited within 2 years of publication and the citations must occur in a 2-month time period. Papers are assigned to 2-month periods and thresholds are set for each period and field to select 0.1% of papers. There were no hot papers identified for the current 2-month period (i.e., March-April 2007), but there were a number of hot papers identified from previous periods.

Using the hot paper thresholds established by ESI as a benchmark, 45 hot papers, representing 2.9% of the PM papers, were identified in six fields—Clinical Medicine, Engineering, Environment/Ecology, Geosciences, Multidisciplinary, and Pharmacology & Toxicology. The number of PM hot papers is 29 times higher than expected. The hot papers are listed in Table 17.

Self-citations are journal article references to articles from that same author (i.e., the first author). Because higher author self-citation rates can inflate the number of citations, the author self-citation rate was calculated for the PM papers. Of the 27,449 total cites, 1,227 are author self-cites—a 4.5% author self-citation rate. Garfield and Sher3 found that authors working in research-based disciplines tend to cite themselves on the average of 20% of the time. MacRoberts and MacRoberts4 claim that approximately 10% to 30% of all the citations listed fall into the category of author self-citation. Kovacic and Misak5 recently reported a 20% author self-citation rate for medical literature. Therefore, the 4.5% self-cite rate for the PM papers is well below the range for author self-citation.

Highly Cited Researchers

A search of Thomson’s ISIHighlyCited.com revealed that 40 (1.5%) of the 2,710 authors of the PM papers are highly cited researchers. ISIHighlyCited.com is a database of the world’s most influential researchers who have made key contributions to science and technology during the period from 1981 to 1999. The highly cited researchers identified during this analysis of the PM publications are presented in Table 18.

National Centers for Environmental Prediction, National Oceanic and Atmospheric Administration

Geosciences

Watson, John G.

Desert Research Institute

Environment/Ecology

Winer, Arthur M.

University of California–Los Angeles

Environment/Ecology

Wolff, George T.

General Motors Corporation

Environment/Ecology

Zeger, Scott L.

Johns Hopkins University

Mathematics

Total = 40

Patents

There were 6 patents issued by investigators from 1998 to 2007 for PM research that was conducted by EPA intramural and extramural researchers. The patents are listed in Table 19. Two of the 6 patents (33.3%) were referenced by a total of 9 other patents.

Table 19. Patents Resulting From PM Research (1998-2007)

Patent or Patent Application No.

Inventor(s)

Title

Patent/Patent Application Date

Patents that Referenced This Patent

U.S. Patent No. 6,890,372

Dasgupta PK Morris KJ
Li J

Denuder assembly for collection and removal of soluble atmospheric gases

1 Thomson Scientific’s Web of Science provides access to current and retrospective multidisciplinary information from approximately 8,830 of the most prestigious, high impact research journals in the world. Web of Science also provides cited reference searching.

2 Scopus is a large abstract and citation database of research literature and quality Web sources designed to support the literature research process. Scopus offers access to 15,000 titles from 4,000 different publishers, more than 12,850 academic journals (including coverage of 535 Open Access journals, 750 conference proceedings, and 600 trade publications), 27 million abstracts, 245 million references, 200 million scientific Web pages, and 13 million patent records.

3 Garfield E, Sher IH. New factors in the evaluation of scientific literature through citation indexing. American Documentation 1963;18(July):195-210.