Wolters Kluwer Health may email you for journal alerts and information, but is committed
to maintaining your privacy and will not share your personal information without
your express consent. For more information, please refer to our Privacy Policy.

Impact of Emergency Preparedness Exercise on Performance

Author Information

Division of Policy Translation and Leadership Development (Drs Agboola, McCarthy, and Biddinger) and Department of Health Policy and Management (Dr Biddinger), Harvard School of Public Health; and Department of Emergency Medicine, Massachusetts General Hospital (Dr Biddinger), Boston, Massachusetts.

The authors acknowledge the funding support from the US Centers for Disease Control and Prevention (CDC) grant number 5PO1TP000307-04 Preparedness and Emergency Response Research Center—Linking Assessment to Measurement and Performance in Public Health Emergency Preparedness Exercises. The content of this publication as well as the views and discussions expressed in this article are solely those of the authors and do not necessarily represent the views of any partner organizations, the CDC, or the US Department of Health & Human Services, nor does mention of trade names, commercial practices, or organizations imply endorsement by the US government. The authors acknowledge the work and commitment of their partners from the Massachusetts Department of Public Health.

The authors declare no conflicts of interest.

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal's Web site (http://www.JPHMP.com).

Abstract

The aim of this study was to investigate whether prior participation in preparedness exercises was predictive of better performance on objective measures of response. We conducted a statewide tabletop exercise that focused on a hazardous materials (HAZMAT) scenario and assessed performance using our developed evaluation tool. The evaluative score was analyzed in relation to the number of HAZMAT exercises in the past 3 years, participation in prior CHEMPACK-specific exercise, hospital size, teaching status of the hospital, preparedness training experience, and participants years of experience in preparedness activities. Hospitals that had participated in more exercises in the past 3 years performed significantly better than hospitals that had participated in fewer exercises. No significant differences were found between the performance of hospitals in relation to size, teaching status, preparedness training experience, and participants' years of experience in preparedness activities. Our results suggest that more frequent participation in exercises may result in improved overall response.

An effective response to public health emergencies and their consequences is crucial to protecting the health of our communities and ensuring stability in access to care. In the last decade, great strides have been made toward the goals of an efficient and sustainable public health emergency preparedness and response system. These strides have been made through on-going reassessments of the potential hazards and risks that health and public health systems may face,1 review of available response resources and refinement of strategies for resource use and allocation,2 enhancement of existing emergency operations plans and systems,3 integration of preparedness training programs into routine practice,4 sharing and incorporation of lessons learned from after-action reports (AARs),5 and development of community education programs.6 As public health and health system preparedness for emergencies have improved, drills and exercises have been frequently recommended both as a means to measure the magnitude of the improvements to their emergency response plans, procedures, and protocols and as a means to identify remaining gaps and challenges. For instance, Centers for Disease Control and Prevention (CDC) requires the Public Health Emergency Preparedness awardees to conduct at least one preparedness exercise per budget period.7 Also, The Joint Commission requires that all hospitals participate in at least 2 emergency preparedness exercises per year as part of their accreditation requirements.8 Critical elements in the design, conduct, and structure of exercises will impact their utility in testing and improving emergency preparedness. It is thought that well-designed exercises can serve as a useful approximation of a real emergency or incident.9–12 Multiple studies have described how participation in these exercises is effective in familiarizing personnel with emergency plans, allowing different agencies to practice working together and revealing flaws in established emergency plans and/or actions of responders.9,12–15 However, little data exists that describes whether participation itself in emergency response exercises can be predictive of performance improvement over time.

Emergency response exercises are not intended to be stand-alone events, disconnected from other preparedness and response efforts. Instead, they are intended to be part of a larger continuous cycle of planning, training, exercising, analyzing shortcomings, and identifying areas requiring improvement, as well as subsequently taking of corrective actions.16 Given the costs of exercising, the importance of integrating exercises into this broader cycle of improving preparedness is crucial. A pilot study conducted by the CDC, for instance, showed that the cost of an exercise for Public Health Emergency Preparedness awardees ranged from $1172 to $22 792.17 A separate study conducted by the Harvard School of Public Health estimated that a regional tabletop exercise involving 5 local health departments with 46 participants would cost about $24 000 for planning, conduct, and evaluation.18 In this study, we endeavored to investigate whether exercise participation itself could be correlated with improved preparedness. In this study, our goal was to determine whether there was an association between participation in prior preparedness exercises and performance on objective measures of response.

Methods

Setting

In collaboration with the Massachusetts Department of Public Health, we developed and conducted a statewide tabletop exercise series between August 2011 and January 2012 in all 6 preparedness regions of Massachusetts.19 A tabletop exercise involves key personnel discussing simulated scenarios in an informal setting based on existing operational plans and identifying where those plans need to be refined.20 The exercises focused on a hazardous materials (HAZMAT) scenario. All of the acute care hospitals in the state, as well as their neighboring public health and emergency medical service providers, were invited to participate.

Exercise design and evaluation process

Participating hospitals in each region played at the same time in the same venue. Hospital-based participants were seated at individual tables with their local public health and public safety partners. Each table was assigned a facilitator to guide the discussion and ensure focus on the salient issues of the exercise. State and regional public safety, emergency management, and public health representatives were grouped at a central table during each of the regional exercise to allow local institutions and responders to communicate with their regional and state counterparts. The facilitators moderated the exercise discussions and posed a set of decision-oriented questions that were developed by the exercise planners. Evaluators compiled notes during real-time exercise play and documented performance using a standardized exercise evaluation tool (see later). The evaluators had an average of 17 years of experience in emergency preparedness (range, 6-32 years), and all had received formal training on incident command system, information management, and HAZMAT. Prior to the exercise, they also received training on the use of the evaluation instrument and scoring guidance and were provided with information regarding the capabilities being tested during the exercise. In the exercise program, we defined hospitals as having sufficient representation if they have at least 4 participants who they considered integral to responding to a HAZMAT event present for the exercise. Therefore, hospitals that were unable to bring sufficient representatives to the exercise did not play independently. These hospital representatives were combined with other representatives in a similar situation during the exercise. Data from hospitals that were unable to participate independently in the exercises were excluded from this study.

Evaluation tool

In previous work, the Harvard School of Public Health has developed an instrument for evaluation of performance in exercises (see Supplemental Digital Content for a copy of the instrument, available at: http://links.lww.com/JPHMP/A24). Our evaluation instrument contains both subjective and objective performance measures aligned in a grid that allows documentation of performance. Each measure contains a checklist of actions and a score so that evaluators can objectively document the performance of specific actions as well as subjectively rate the quality of such performance. The measures are selected on the basis of the exercise objectives. Data are collected using both checklists of expected actions for each measure and completion of a 10-point Likert scale for each measure with responses ranging from 1 (very poor) to 10 (very good) (see Supplemental Digital Content for details, available at: http://links.lww.com/JPHMP/A24). The evaluation instrument has been pilot-tested for its reliability, usability, and validity by independent evaluators during 5 tabletop exercises. The interrater reliability ranged from moderate to substantial agreement (k = 0.52-0.68; P < .001). Ranking of the entities that participated in these exercises using the evaluation score from the instrument was consistent with what was originally hypothesized by a group of experts. Recommendations from content experts and evaluators were used to modify the instrument. In addition to collecting specific data on the chosen measures, evaluators document unique observations in their evaluation forms and also offer responses to open-ended questions at the end of the instrument. For this exercise, the evaluation instrument consisted of 23 measures. The measures selected for this exercise program were identified through a comprehensive review of literature based on the exercise objectives, review of prior valid measures identified by our research program, and recommendation by content experts. The measures were then grouped into 3 subsections to correlate with exercise play: (1) ability to maintain situation awareness, receive appropriate notifications, and act on incident information (8 tasks); (2) ability to request, activate, and receive/transport CHEMPACK (6 tasks); and (3) ability to decontaminate, triage, and manage contaminated or potentially contaminated patients (9 tasks). CHEMPACK is part of the CDC's Strategic National Stockpile program and is designed to preposition antidotes to expedite the treatment of individuals exposed to chemical nerve agents.21

Data collection and analysis of performance

Individual evaluators transferred their data from their handwritten forms into an online data collection form designed by our program to facilitate electronic submission. Evaluation data were then downloaded from the online version of the evaluation tool into Microsoft excel database. A point was allocated for each checklist of action that was accomplished (n = 87), and a quantitative score between 1 and 10 was allocated for every measure (maximum value = 230). The total performance score was calculated by summing the checklist of action score and the quantitative score (maximum value = 317). The open-ended questions were used to validate the scores and to prepare the AAR on the exercise.

Variable of interest

To try to correlate performance data with hospital characteristics that might also affect exercise performance, we analyzed the evaluative score in relation to the following selected variables of interest: (1) mean number of hospital HAZMAT exercise attended by participants of each hospital in the past 3 years—this was used as a proxy for the hospital HAZMAT experience of each hospital; (2) hospitals participation in prior CHEMPACK-specific exercise; (3) teaching status of the hospital (teaching hospitals vs nonteaching hospital); (4) hospital size (small hospitals, <180 beds; medium-sized hospitals, 181-339 beds; and large hospitals, ≥340 beds); (5) mean number of preparedness training attended by participants of each hospital in the past year and; (6) mean years of emergency preparedness and response experience of participants of each hospital.

Statistical analysis

Data were analyzed using STATA 12.0. Descriptive analysis was used to assess the performance. The total performance scores were normally distributed as tested by a normal probability plot and skewness/kurtosis test; therefore, parametric statistical methods were used for comparison between subgroups. A 2-sided t test was used to compare means of exercise performance between hospitals with respect to the number of prior HAZMAT exercise, participation in CHEMPACK exercise, and teaching status of the hospital. Univariate analysis of variance was used to compare means of exercise performance with respect to hospital size. Spearman ρ coefficient was used to assess the correlation between exercise performance and other variables of interest.

RESULTS

Background characteristics

Of 74 acute care hospitals in Massachusetts, 47 (65%) hospitals participated in the exercise. In all, 22 (47%) of the 47 participating hospitals were deemed to have insufficient participants to fully participate in the exercise independently. These hospitals were gathered into groups with other similar hospitals for exercise play, but they were excluded from our data analysis because they were evaluated as a group rather than individually. Twenty-five (53%) hospitals represented by 170 participants were therefore included in our analysis. Of the hospitals that were included in the analysis, 11 (44%) were small, 4 (16%) were medium-sized, and 10 (40%) were large. Twenty (80%) hospitals were nonteaching hospitals, whereas the remaining 5 (20%) were teaching hospitals. In terms of the number of prior exercise participation, 11 (44%) hospitals had participated in 2 or fewer HAZMAT exercises in the past 3 years (mean number of exercises = 1.7). Fourteen (56%) hospitals had participated in 3 or more exercises in the past 3 years (mean number of exercises = 4.9). Four (16%) hospitals participated in the prior CHEMPACK exercise, whereas 21 (84%) hospitals did not. On average, exercise participants had attended 3 emergency preparedness trainings in the past year and their average duration of experience in emergency preparedness was 11.6 years.

Impact of hospital and participants' characteristics on exercise performance

Number of prior HAZMAT exercises in the past 3 years

The overall performance score of hospitals that had participated in 3 or more exercises in the past 3 years was higher (mean = 67%; range, 50%-85%) than those that had participated in fewer exercises (mean = 55.7%; range, 42%-62%). The performance difference between the 2 groups was significant (P = .004). This difference persisted, even when analyzed by each of the subsection of the exercise (Table).

Participation in prior CHEMPACK exercises

By coincidence, some hospitals in our exercise program had participated in a different response exercise program focused solely on the CHEMPACK program within the past 6 months. We performed a separate analysis of the data contrasting those hospitals that had recently tested their CHEMPACK plans with those that had not. The performance score of hospitals that had participated in the CHEMPACK exercise that took place about 6 months prior to this exercise was 62%, whereas those that did not participate in the CHEMPACK exercise had a slightly higher score of 63%. The difference was not statistically significant (P > .05).

Size of hospitals

The mean performance score of medium and large hospitals was the same (66%), whereas that of the smaller hospitals was lower (mean = 57%). A one-way analysis of variance test showed that the differences among the sizes of the hospitals were not statistically significant (P > .05).

Emergency preparedness training attended in the past year

The correlation between the mean number of emergency preparedness training attended by participants of a hospital and the performance score was low (r = 0.2; P > .05). Hospitals with an average number of training of 3 or more (mean score) scored an average of 58%, whereas hospitals with an average number of training of fewer than 3 (mean score) scored an average of 59%.

Discussion

Exercises are commonly used by health sector entities to test emergency preparedness and response capabilities. It is well documented that such exercises can uncover gaps in planning, conflicts with other plans, inadequate resources, and other problems.9,12–15 It is less known, however, whether the act of participating in exercises itself assists in improving emergency preparedness and response. In this study, we endeavored to assess the relationship between repeated exercising and the overall emergency response capabilities of hospitals, as measured by performance during a tabletop exercise.

Our study shows that hospitals that participate in emergency response exercises more frequently perform better on a standardized assessment tool than those that exercise less frequently. There was, however, no correlation between performance and hospital size and teaching status. Djalali et al22 obtained similar findings when they assessed the relationship between performance during exercise as measured by indicators of hospital incident command system and hospital size. Interestingly, performance was also not correlated with years of experience with emergency preparedness of the hospital staff participating in the exercise or the number of recent trainings they had attended. These data suggest that participation in previous exercises and subsequent improvement of emergency plans and systems are more likely the driver of better emergency response performance.

One might hypothesize that the improved performance observed in participants who exercised more frequently was the result of their having “taken the test” multiple times and therefore knowing the “right answers” more often. We believe this is not the case. First, none of the hospitals had participated in this exercise scenario previously and therefore had not known what to expect. Second, the improved performance was seen in only the 2 phases of exercise play that focused on emergency plans that are under local control. The first and third phases of our exercise tested notification/communication systems and on-site decontamination plans respectively, which are under the control of the individual hospitals. The second phase of our exercise tested aspects of the CHEMPACK plan, which is not written by local entities or under their control. Therefore, it is not surprising that hospitals that exercised more frequently were unable to improve performance on a phase not under their control (CHEMPACK). This relation between lack of difference in observed performance and ability to control plans is underscored by the subset analysis we were able to perform. Although our sample size is small, those hospitals who recently exercised their CHEMPACK plans did not seem to demonstrate improved performance when compared with those that did not exercise those plans recently. We believe that this lack of improvement may be the result of a lack of ability to amend those plans and make local improvements. This observation suggests that the observed differences in performance are not the result of taking the test multiple times but may be the result of implementation of local improvements by hospitals after exercising. In the 2008 National Profile, 76% of the local health departments that participated in exercises reported that they revised their written emergency response plans on the basis of recommendations from an exercise AAR.23 A review of the AAR of the response to the 2011 Joplin, Missouri, Tornado also showed how participation in National Level exercise and other periodic exercises helped federal, state, regional, local, and private sector personnel respond effectively to the Joplin Tornado.24 We believe that the improved performance seen in hospitals that exercise more frequently is likely the result of greater iterative identification of gaps and problems in emergency response and iterative correction of those gaps and problems. Therefore, independent of the scenario, hospitals that exercise more frequently might be expected to generally perform better than those that do not on aspects of emergency response that are under their local control. On the basis of other studies, we also hypothesize that this observed improvement in exercise performance is also likely to translate into improved performance in response to real-world emergencies25,12; however, our data do not address this issue.

Limitation

Our study had several important limitations. First, we were only able to observe the performance of hospitals in one exercise scenario. Our findings would be stronger if they were found to persist over multiple exercises and multiple scenarios. Second, although we collected data from a statewide exercise series sponsored by the state Department of Public Health, not all hospitals in the state were able to participate. Furthermore, some hospitals were unable to send sufficient participants to the exercise to meaningfully represent their institution and were therefore excluded from this analysis. The resulting smaller sample size resulted in reduced power. It is possible that our sample size was simply too small to detect an association between performance and some variables. A larger sample could provide more definitive evidence and would allow for greater statistical analysis. The size also limits the generalizability of this study. Third, the mean number of the participants' hospital HAZMAT exercise experience (over a 3 year period) from each table was used as a proxy for the HAZMAT exercise experience of each hospital. We believe that this gave us a good estimate of the experience of each hospital because the analysis was limited to the hospitals that had sufficient representation, that is, having at least 4 participants present for the exercise who they considered integral to responding to any HAZMAT event. Fourth, we did not take into account the types of exercises conducted or the types of trainings held. There are clearly differences in exercises and trainings that may have had an impact on the resulting improvement of the participants. Finally, the method we used to calculate the total evaluation score from the instrument has not yet been validated. Therefore, this may not give us an accurate estimate of the individual hospital performance. However, we feel that the estimate gives an indication of how well the hospitals performed relative to each other. There are several possibilities for future research. In addition to increasing the sample size and looking at different exercise scenario, future research should analyze the effect of participation in exercises on real-world response.

Conclusion

Exercises to test preparedness are frequently used to test capabilities and response plans in the face of various scenarios. Little is known about how participation in exercises affects preparedness and response in other scenarios. Our data show that more frequent participation in emergency preparedness exercises is correlated with improved performance measured in a tabletop exercise. Performance appears to be unrelated to hospital size, teaching status, the years of experience of the participants in preparedness, and the number of trainings participants recently attended. This suggests that more frequent participation in exercises may result in improved overall response. This appears to support current recommendations for exercising frequently7,8 to ensure optimal readiness and response.

Enter and submit the email address you registered with. An email with instructions to reset your password will be sent to that address.

Email:

Password Sent

Link to reset your password has been sent to specified email address.

Remember me

What does "Remember me" mean?
By checking this box, you'll stay logged in until you logout. You'll get easier access to your articles, collections,
media, and all your other content, even if you close your browser or shut down your
computer.

To protect your most sensitive data and activities (like changing your password),
we'll ask you to re-enter your password when you access these services.

What if I'm on a computer that I share with others?
If you're using a public computer or you share this computer with others, we recommend
that you uncheck the "Remember me" box.