Wolters Kluwer Health
may email you for journal alerts and information, but is committed
to maintaining your privacy and will not share your personal information without
your express consent. For more information, please refer to our Privacy Policy.

Purpose: Medical educators use key features examinations (KFEs) to assess clinical decision making in many countries, but not in U.S. medical schools. The authors developed an online KFE to assess third-year medical students’ decision-making abilities during internal medicine (IM) clerkships in the United States. They used Messick’s unified validity framework to gather validity evidence regarding response process, internal structure, and relationship to other variables.

Method: From February 2012 through January 2013, 759 students (at 8 U.S. medical schools) had 75 minutes to complete one of four KFE forms during their IM clerkship. They also completed a survey regarding their experiences. The authors performed item analyses and generalizability studies, comparing KFE scores with prior clinical experience and National Board of Medical Examiners Subject Examination (NBME-SE) scores.

Results: Five hundred fifteen (67.9%) students consented to participate. Across KFE forms, mean scores ranged from 54.5% to 60.3% (SD 8.4 9 – 9.6%), and Phi-coefficients ranged from 0.36 to 0.52. Adding 5 cases to the most reliable form would increase the Phi-coefficient to 0.59. Removing the least discriminating case from the two most reliable forms would increase the alpha coefficient to, respectively, 0.58 and 0.57. The main source of variance came from the interaction of students (nested in schools) and cases. Correlation between KFE and NBME-SE scores ranged from 0.24 to 0.47 (P < 0.01).

Conclusions: These results provide strong evidence for response-process and relationship-to-other-variables validity and moderate internal structure validity for using a KFE to complement other assessments in U.S. IM clerkships.

V.J. Lang is associate professor of medicine, director, Medicine Sub-Internship, and senior associate division chief, Hospital Medicine, University of Rochester School of Medicine and Dentistry, Rochester, New York.

N.B. Berman is professor of pediatrics and of medical education, Geisel School of Medicine at Dartmouth, Hanover, New Hampshire.

K. Bronander is professor of medicine and medical director, Simulation, University of Nevada, Reno School of Medicine, Reno, Nevada.

H. Harrell is professor of medicine and co-director, Medicine Clerkship, University of Florida, Gainesville, Florida.

S. Hingle is professor of medicine, director, Year 3 Curriculum, and director, Faculty Development, Southern Illinois University School of Medicine, Springfield, Illinois.

Funding/Support: Project support was provided by Aquifer Internal Medicine (Hanover, New Hampshire), a non-profit organization that distributes the key features examination described in this report.

Other disclosures: Dr. Valarie J. Lang, Dr. Kirk Bronander, and Dr. Heather Harrell are supported for their work with Aquifer Internal Medicine (Hanover, New Hampshire), a non-profit organization that distributes the key features examination described in this report.

Ethical approval: The authors procured institutional review board approval to conduct the study described in this report from Case Western Reserve University, University of Florida, Indiana University School of Medicine, University of Louisville, University of Nevada at Reno School of Medicine, University of Rochester School of Medicine & Dentistry, Southern Illinois University, University of Illinois at Chicago, and University of Illinois at Peoria.

Disclaimers: The work presented here is that of the authors and does not necessarily represent the views or policy of the United State Veterans Affairs Administration.

Previous presentations: Portions of the study were presented at Academic Internal Medicine Week (New Orleans, Louisiana, October 2013); the Ottawa Conference (Ottawa, Canada, April 2014); and the University of Illinois at Chicago Masters in Health Professions Education conference (Chicago, Illinois, July 2015).