Appendix II: Quantitative Analysis Methodology

During the process of supporting the PSW working group, in particular the PSW data subcommittee, Thomson Reuters designed and built a reporting system that facilitated the many data requests from the subcommittees. A first step was to work closely with the PSW working group and NIH staff to define the analysis requirements and identify data sources. Once the available data was collected and centralized in a database, the focus turned to the preferred delivery format. An automated reporting module was developed, one that queries the data system and produces tables and charts to committee specifications. The intent was to avoid a one-time, one-report process in favor of a data system that can facilitate future tracking and evaluation of any recommendations implemented by the NIH and other organizations in response to the findings of the PSW working group.

A key feature of the PSW data analysis is how it differs from standard data reporting at NIH, which normally focuses on reporting application and award total counts and breakdowns. Instead, PSW’s unit of analysis is the individual: Each individual is tracked by the data system throughout the PSW pipeline to create a history of research outcomes. Hence, trends can be established for different groups of people (not just programs), and unique career tracks can be examined more carefully and compared to produce a better picture and understanding of the physician-scientist workforce.

Data Sources

Unless otherwise noted, the source of data for all charts and tables included in this report are from NIH’s IMPACII data system, supplemented with AAMC Faculty Roster data, as provided under a data sharing agreement with AAMC. Aggregate data on faculty and physician-scientists were provided by the American Medical Association (AMA), the American Dental Education Association, the American Veterinary Medical Association and the Association of American Veterinary Medical Colleges.

Select reports were generated using data from the NIH Medical Scientist Training Program (MSTP) and summary data from the AAMC’s Matriculating Student Questionnaire and Medical School Graduation Questionnaire. Aggregate data on faculty and physician-scientists were provided by the American Medical Association (AMA), the American Dental Education Association, the American Veterinary Medical Association and the Association of American Veterinary Medical Colleges.

In addition, data analyses were carried out with significant support form NIH’s Division of Statistical Analysis and Reporting (DSAR) within the Office of Extramural Research. Specifically, DSAR staff provided data on T32 appointees’ outcomes and other data review and analysis.

The NIH awards and time period selected for inclusion in the system from IMPACII were:

As noted above, a key requirement for this analysis was to track individuals throughout the PSW pipeline at key career stages (i.e., a first-time RPG applicant RPG or, say, R01). Accordingly, a method was devised by Thomson Reuters to count individuals and their associated applications and awards on any given year during the time-period.

An individual applicant was defined as: in a given Fiscal Year, an individual applicant is counted only once per IC and mechanism. In the event an applicant applies more than once in a given FY to an IC and mechanism, the most recent awarded application is selected.

The approach used by the PSW data subcommittee differs from standard data reporting at NIH, which normally focuses on reporting application and award total counts and breakdowns. These differences may lead to discrepancies when comparing trends presented here to those reported for applications and awards.

Degree categories and assigned groupings (individuals could have more than one degree) were reviewed by the PSW data subcommittee, and the degree categories were set as: