How the NEJM Knowledge+ Team Developed Our Pediatrics Board Review Questions

Whether in training or in practice, whether preparing for clinic or for a board exam, pediatricians are in need of efficient ways to grow and maintain their knowledge. Last week NEJM Group announced the launch of NEJM Knowledge+ Pediatrics Board Review. The pediatrics board review questions provide a wide variety of clinical scenarios with challenging questions on every aspect of pediatric care, from prevention and screening to diagnosis, evaluation, and management. Every question undergoes intensive, thoughtful review by a team of pediatricians with both clinical and teaching experience. The topics reflect the American Board of Pediatrics content outline, and each question provides detailed feedback and useful links to additional resources. The combination of these pediatric board review questions and the adaptive algorithm that creates a unique pathway for each learner yield a powerfully engaging way to master the underlying learning and ensure that your self-assessment of your knowledge is accurate.

How We Developed the Pediatrics Board Review Questions

The NEJM Knowledge+ editorial department recruited authors from across the country to write our pediatrics board review questions. Our question authors were fellows and faculty at the following institutions:

Children’s Hospital of Philadelphia

Cincinnati Children’s Hospital Medical Center

Massachusetts General Hospital

University of Michigan Hospital and Health Systems

Vanderbilt University

Washington University

Each authoring site wrote their own cases, questions, answer options, and detailed feedback; authors split up the questions among a small group and filtered drafts to a site director after refining them among themselves. The NEJM Knowledge+ editorial department provided training materials and sample questions as well as access to NEJM Knowledge+ for internists and family practitioners — this way they could get a sense of the kinds of questions they would be writing for Pediatrics Board Review.

We (Deborah Lehman and Shannon Scott-Vernaglia) were the first to review and edit the questions as they were submitted to NEJM. Here’s a little bit of background about us:

Deborah Lehman, MD, is a clinical professor of pediatrics at the David Geffen School of Medicine at the University of California, Los Angeles, and the associate vice chair for education in the department of pediatrics. She is the pediatric clerkship site director for third-year medical students and provides oversight to the subspecialty fellowship programs in the department. She is also on the board for NEJM Journal Watch Pediatrics and Adolescent Medicine, providing pediatric infectious disease summaries.

Shannon E. Scott-Vernaglia, MD, is a practicing general pediatrician and the program director for the Pediatrics Residency at Massachusetts General Hospital. Her educational interests include simulation, reflective practice, and faculty development. She is active in the Association of Pediatric Program Directors’ Faculty Development Task Force, and she spearheads faculty development for the Pediatrics Department at MGH.

Our role as first-line reviewers and editors of the NEJM Knowledge+ pediatrics board review questions involved a multistep approach. First, we wanted to be sure each clinical vignette seemed realistic and provided a robust background for the knowledge to be assessed. We then wanted to ensure that answer choices were plausible and yet challenging to the reader. We particularly sought out questions that involved the learner making a most likely diagnosis, but then responding to a second order question, perhaps about patient management or associated findings. Reading the detailed feedback is how the learner is really going to solidify his or her knowledge, making sure a correct answer was made for the “right” reasons and understanding why an incorrect choice was incorrect. We required that each answer choice be addressed in the feedback to maximize learning.

After the questions go through sometimes multiple rounds of editing and revision, they go to a panel of specialists, generalists, and editors who refine them further. At any point, the questions can return to an earlier stage in the process. Even after reaching the final stage and being published in NEJM Knowledge+, questions will be revised with changes in medical guidelines as well as in response to feedback from learners. We keep track of updates to the content here.

Types of Questions

NEJM Knowledge+ has three types of pediatrics board review questions:

Case-based multiple choice

Short multiple choice

Fill in the blank

All three types test you on the same learning objectives. For example, here are two related learning objectives from our Pediatrics Board Review product:

Recognize the clinical features of volvulus that is secondary to intestinal malrotation in an infant.

Choose the most appropriate management for an infant with volvulus that is secondary to intestinal malrotation.

The case-based question would test both learning objectives, while the short question might only test the second one or might test both. The fill-in-the-blank question would test one or the other, but not both.

Case-based multiple-choice questions always have a full patient vignette describing the age and sex of the patient, the medical history (if remarkable), current symptoms and vitals, and sometimes a full-color photograph of a skin finding; a table of laboratory findings; an image of an electrocardiogram, radiograph, CT scan, or other diagnostic test; or all of the above.

Short multiple-choice questions often include the same key information as the case-based questions, but they often succinctly summarize the vignette and laboratory findings rather than present them as a narrative.

Our multiple-choice questions have five answer choices: one correct answer and four “distractors.” All four incorrect answers need to be plausible but clearly unjustifiable in the situation described in the case. All the information a learner needs to answer correctly is included in the case question or inside their own memory bank.

Fill-in-the-blank questions are based on the key learning points for each question and require that the learners improve their recall: there are no answers to choose from. The editorial staff ensures that synonyms are accepted by the system — physicians often abbreviate commonly used terms.

Relevance to Residents and Physicians in Practice

Residents routinely ask faculty and program directors for guidance on preparing for the Pediatrics Certifying Examination and frequently the reply is “practice doing questions.” The questions in NEJM Knowledge+ Pediatrics Board Review are on the topics you are likely to see on the pediatric boards as well as patient scenarios you will see in the real world.

How We Chose — and Verified — References

The references that we link to on every question in NEJM Knowledge+ must support the correct answer and the detailed feedback for that learning objective and the multiple-choice questions associated with it. In some cases, a less recent review article does a fantastic job detailing the diagnosis and treatment scenarios of a given condition that has not changed in some time, but sometimes a more recent study has changed physician practice and we will provide both references for the learner.

We chose references that appear in well-respected journals among academics and pediatricians at large, including Pediatrics, Pediatrics in Review, some subspecialty literature, and of course The New England Journal of Medicine. In addition, we link to free online sources, such as Centers for Disease Control and Prevention and the World Health Organization, plus up-to-date guidelines so that you can stay current in your medical knowledge.

Our specialist section editors reviewed every reference choice and made sure the information provided in the references remains relevant and supports the question’s correct answer. Our final review panel of generalists also look at the entire question set, detailed feedback, and citations to be triple-sure that the references are applicable and not outdated. Finally, learners using NEJM Knowledge+ let us know via the Challenge Us button if they have seen a more recent study or review article that contradicts our references or if they have a question or comment about anything else in the question or feedback.

Makeup of the Pediatrics Board Review Question Bank

Using the ABP Content Outline

We chose to align our pediatrics board review questions with the American Board of Pediatrics content outline (also known as a blueprint). The ABP recently updated their list of topics that will be covered on the boards (initial certification as well as maintenance of certification), so we also updated our list of modules and topics in NEJM Knowledge+.

Some of the topics lent themselves to being combined for easier navigation on the dashboard and reports page in the product.

The editorial team considered which topic areas were most important to residents and physicians in practice and the site directors at each authoring institution outlined learning objectives under those topic headings. We used the ABP content outline as a guide during this process, because of its detailed list of topics and learning objectives — adding further dimensions to the case vignette to form the basis of our question sets. The new content outline, which takes effect in September 2017, does not include learning objectives, so we were excited to utilize the level of detail in the previous content outline for designing our learning objectives.

Other dimensions of the case and question that we had our authors and editors consider when writing their multiple-choice questions include patient age and sex, clinical setting, chronicity, decision focus, and ACGME core competency.

Case vignettes are presented in a variety of settings to reflect real practice, but core concepts tested are those general pediatricians are expected to understand.

Writing Questions for an Adaptive Learning Platform

In this excellent interview with Area9 Learning’s founder, Ulrik Christensen, you’ll read a description of what an adaptive learning program for physicians should do: “NEJM Knowledge+ will guide people through all the hard work they need to do in a very structured, fact-driven, and efficient way.”

Writing multiple-choice questions for an adaptive learning system is a bit different from writing questions for a static question bank, because the questions will be returned to the learner to lock in the knowledge over time. The act of self-assessment is the education. This is called formative assessment, and it is a powerful concept — the cornerstone of effective adult learning. Frequent repetition of what we need to know is clearly the most efficient way to retain knowledge.

In this program, we aim to create a question bank that covers not just a sample of the learning objectives from the ABP blueprint but a question bank that covers the core knowledge a physician needs to know to excel in daily practice. If we’ve succeeded in this, then learners will feel (as the system adapts to them) that the questions they find easy are not being returned to them until some time has passed and that questions they struggled with are returned to them at intervals, in order to combat memory decay.

Recharge, the personalized review program in NEJM Knowledge+, takes into account how long it has been since learners have seen a question and whether or not they answered with confidence. As learners read the key learning points, detailed feedback, and citations, they should feel that they are grasping more and more of the material they need to recall for the boards and in their practice. The way we do this is to write granular learning objectives and questions that require cognition, skill, and various levels of understanding of the material.

• •

Developing accurate, relevant, and thoughtful cases and questions is no small task. From blueprint development to author selection and then through the many layers of review, we make sure that every question matters.

Leave a Comment

Note: This is a moderated forum. By clicking on the "Submit Comment" button below, you agree to abide by the NEJM Knowledge+ Terms of Use. We reserve the right to not post every comment. Read our Comments Policy for more details.