Abstract

Background Reaction time (RT) is a valuable component of the sport concussion assessment battery. RT is typically measured using computers
running specialised software, which limits its applicability in some athletic settings and populations. To address this, we
developed a simple clinical test of RT (RTclin) that involves grasping a falling measuring stick.

Purpose To determine the effect of concussion on RTclin and its sensitivity and specificity for concussion.

Materials and methods Concussed athletes (n=28) and non-concussed control team-mates (n=28) completed RTclin assessments at baseline and within 48 h of injury. Repeated measures analysis of variance compared mean baseline and follow-up
RTclin values between groups. Sensitivity and specificity were calculated over a range of reliable change confidence levels.

Results RTclin differed significantly between groups (p<0.001): there was significant prolongation from baseline to postinjury in the concussed
group (p=0.003), with a trend towards improvement in the control group (p=0.058). Sensitivity and specificity were maximised
when a critical change value of 0 ms was applied (ie, any increase in RTclin from baseline was interpreted as abnormal), which corresponded to a sensitivity of 75%, specificity of 68% and a 65% reliable
change confidence level.

Conclusions RTclin appears sensitive to the effects of concussion and distinguished concussed and non-concussed athletes with similar sensitivity
and specificity to other commonly used concussion assessment tools. Given its simplicity, low cost and minimal time requirement,
RTclin should be considered a viable component of the sports medicine provider's multifaceted concussion assessment battery.

Introduction

Sport-related concussion (SRC) is common1 and there is increasing concern regarding possible long-term effects.2–4 While concussed athletes typically recover symptomatically within 1–2 weeks,5,6 an athlete whose concussion is unrecognised or who returns to play prematurely is put at risk for more significant injury.7 Thus, prompt recognition of SRC and accurate determination of its resolution are important. However, SRC remains a diagnostic
challenge. Presentations are variable and often subtle, and athletes may minimise their symptoms.8 While a growing number of assessment tools have been developed, the diagnosis of SRC remains a clinical one.

Current consensus opinion advocates a multifaceted approach to concussion assessment.9–11 Clinical tools are available to evaluate many aspects of concussion including symptoms,12 neurocognitive function13 and balance.14 Computer-based neurocognitive assessment tools capable of measuring reaction time (RT), attention, working memory and problem
solving are also available.15–17 Such tools can be valuable, but their computer-dependence, time requirement and cost limit their applicability in some situations
(eg, acute concussion diagnosis) and environments (eg, communities with limited financial resources).

A prolonged RT is common following SRC, and is one of the most sensitive indices of neurocognitive change following injury.18,19 In addition RT has been shown to have prognostic value20 and commonly parallels other concussion symptoms.21–24 Moreover, a prolonged RT can persist beyond symptom resolution, suggesting incremental value over solely tracking postconcussive
symptoms.5,25,26

Despite these advantages, computer-independent methods for assessing RT are not available. To address this, we developed a
simple, inexpensive method for clinically assessing RT, referred to as clinical RT (RTclin). RTclin is a visuomotor test in which the subject arrests a falling object by hand closure after it is released by an examiner. Pilot
work demonstrated the short-term and long-term reliability of RTclin,27,28 and its validity with relation to a computerised measure or RT in athletes.29 Furthermore, the task appears to be intrinsically motivating,30 an important characteristic for situations in which an athlete's after-injury performance is compared to preseason baseline.31 Unlike other tests of RT, RTclin has demonstrated functional relevance by showing a strong correlation with an athlete's ability to protect their head in
a laboratory simulated athletic environment.32 Recently, we reported a 13.5% prolongation of RTclin in nine collegiate athletes tested within 72 h of SRC as compared with baseline.33 Although the results are suggestive, a larger, controlled study is needed to further elucidate the clinical usefulness of
RTclin. Therefore, the purpose of this study is to determine the effect of concussion on RTclin in a larger sample of high school and collegiate athletes through a controlled study design. We hypothesised that RTclin would be prolonged in concussed athletes as compared to baseline and non-concussed control athletes.

Methods

Subjects

Male and female athletes participating in multiple sports at two universities and one high school were recruited during preparticipation
physical examinations over two consecutive seasons. Potential subjects were excluded if they were recovering from a concussion
or had an upper limb injury preventing the completion of the RTclin task. Adult athletes provided institutional review board (IRB) approved informed written consent and minor athletes provided
IRB-approved assent with accompanying parental consent. For the purposes of this study, concussion was diagnosed by the athlete's
treating physician as defined in the 2008 Zurich Consensus Statement on Concussion in Sport.9

Testing schedule

RTclin was measured in all athletes at baseline during preparticipation examinations. RTclin was determined again within 48 h of injury in athletes who sustained a physician-diagnosed concussion. For each injured athlete,
a control athlete was who was a member of the same team and was present in the training room for a reason other than concussion
was also retested. Athletes with a dominant upper limb injury preventing completion of the RTclin task were again excluded, analogous to during baseline testing.

Measuring RTclin

The protocol for RTclin testing has been previously described28,29 and is illustrated in figure 1. The RTclin apparatus is an 80 cm rigid measuring stick coated in friction tape with a weighted rubber disk affixed to one end. Athletes
sat with their dominant forearm resting on a table with their hand positioned over its edge. The hand was open surrounding,
without touching, the weighted disk portion of the RTclin apparatus. The examiner suspended the device vertically such that the top face of the disk was aligned in the plane defined
by the top of the athlete's open hand. After predetermined randomly assigned delays ranging from 2 to 5 s, the examiner released
the apparatus and the athlete caught it as quickly as possible by hand closure. For each trial, an RTclin value (in ms) was calculated from the distance (in cm) that the device fell using the formula for a body falling under the
influence of gravity (d=0.5 gt2). After two practice trials, each athlete completed eight data acquisition trials and a mean RTclin value was calculated for the test session. The mean RTclin value was used for analysis. RTclin change scores were determined by the difference between baseline and follow-up values (follow-up RTclin−baseline RTclin) so that a positive RTclin change score represents a decline from baseline, while a negative score represents improvement.

Data analysis

A two-way repeated measures analysis of variance, with paired t tests used post hoc, was used to compare preseason and after-injury
RTclin values in concussed versus control athletes. The effect sizes of the RTclin changes observed were described using Cohen's d. Reliable change calculations were performed using SDs of difference scores
(SDdiff) from the control group.34 An adjusted reliable change equation, controlling for practice effects,35 was used:
where x2−x1 is the individual performance difference between follow-up (x2) and baseline (x1) RTclin in a concussed athlete; μ2−μ1 is the mean group difference between follow-up (μ2) and baseline (μ1) RTclin in the control group; and SDdiff is the SD of the change scores for the control group. An adjusted reliable change index has been demonstrated to perform
comparably to more complex regression-based formulas36–38 and has been advocated over regression-based methods.36

Reliable change values can be interpreted by comparing them to the z-distribution critical value associated with a desired
one-tailed probability of error. One-tailed critical values were used because the outcome of interest was a decline in performance
from baseline, rather than any difference, which could represent either decline or improvement.39,40 For example, a critical value of 1.65 is used when a 5% type I error rate is deemed acceptable. Therefore a reliable change
score greater than 1.65 can be interpreted as an indication that an athlete has experienced a significant decline in their
RTclin performance, with a 5% probability of type I error.

Conversely, once the mean RTclin change score and its associated SD are calculated in the control group, a cut-off value can be generated indicating the RTclin change score that represents significant change at a given level of confidence. We calculated reliable change cut-off values
over a range of confidence levels and determine the combined sensitivity and specificity associated with each value.41 In addition, a receiver operating characteristic curve (ROC curve) was generated to compare the sensitivity and specificity
associated with cut-off value over the range of RTclin change scores observed. Sensitivity and specificity values were summed at each cut-off value, with the highest summed score
interpreted as having the greatest combined sensitivity and specificity.41,42

Results

Twenty-six athletes sustained 28 concussions (with two athletes sustaining repeat concussions) over the 2-year study. The
majority of concussions occurred in male collegiate football players (n=20, 71.4%). The remaining injuries were sustained
by athletes participating in male high school football (n=2, 7.1%), female collegiate soccer (n=2, 7.1%), male collegiate
ice hockey (n=1, 3.6%) and male high school ice hockey (n=1, 3.6%). Of note, two injuries which occurred during year 2 involved
athletes who did not have baseline RTclin results available from that year. Therefore, year 1 baseline RTclin results were used for these subjects and for their matched controls. Eight of the postconcussion testing sessions were completed
on the day of injury, 12 were completed 1 day after injury and eight were completed 2 days after injury not exceeding 48 h.
All concussed athletes were still reporting postconcussion symptoms at the time of testing. The time from baseline to follow-up
RTclin testing did not differ between the concussed and control groups (mean±SD=60.5±40.5 and 73.8±51.0 days, respectively; p=0.631).

Reliable change cut-off values for RTclin and the associated sensitivity and specificity at each confidence level are presented in table 1. The cut-off values represent the RTclin change values necessary to conclude that an athlete has experienced a significant prolongation in RTclin at a given level of confidence. An ROC curve comparing the sensitivity and specificity of RTclin at every possible cut-off value over the observed range of RTclin change values was also generated (figure 3). Resulting analysis demonstrated that cut-off values from 0 to −2 ms (ie, 2 ms of improvement at follow-up) maximised sensitivity
and specificity of RTclin with a combined value of 1.43. This corresponds to a critical value of 0.375 and an associated one-sided confidence level
of 65%.

Reliable change cut-off values for RTclin at a variety of confidence levels with the number of concussed and control subjects who demonstrated greater decline than
the cut-off value as well as the corresponding sensitivity and specificity of the RTclin test at each confidence level

Discussion

In this study we observed a significant decline in RTclin performance following SRC in a group of 28 concussed athletes (8.4% slower RT). These results are similar to those previously
reported.33 The effect size reported here is generally considered to be moderate in size.43 In contrast, follow-up RTclin performance showed a trend towards improvement in the control athletes. Group analyses showed that the contrast between postinjury
performance declines observed in the injured athletes versus the improvement in controls was highly significant.

Reliable change analyses allow group data to be applied to individuals. We found that to be 95% confident an athlete's RTclin performance has declined, RTclin must be at least 31 ms slower during follow-up testing. Using this cut-off value correctly identifies 11 of the 28 concussed
athletes studied (39% sensitivity), but it incorrectly identifies just 2 of the 28 control athletes as having performed worse
(93% specificity). Conversely, if we choose to be 60% confident that an athlete's follow-up RTclin performance is worse the cut-off value is 3 ms improvement from baseline, due to the non-concussed athletes having faster
RTclin at follow-up. These control athlete data suggest a small learning effect, consistent with prior observations.27 Using this 3 ms improvement cut-off value correctly identifies 22 of the 28 concussed athletes (79% sensitivity), while incorrectly
identifying 11 of the 28 control athletes (61% specificity).

Screening tests should have high sensitivity so as to identify the majority of affected individuals. Given the inherent trade-off
between sensitivity and specificity, an increased sensitivity invariably results in an increased false-positive rate. However,
given the current ‘when in doubt sit them out’ approach with regard to concussion, this trade off seems appropriate. This
is especially true in younger athletes, where the consequences of a missed injury are greatest.7 Moreover, maximising test sensitivity at the expense of specificity has been advocated by other groups with regard to concussion
assessment tools.42,44–47 In our study, a change score of 0 ms (ie, interpreting any follow-up RTclin value, that is, slower than baseline as abnormal) was found to maximise the combined sensitivity (75%) and specificity (68%)
of the test and correlated with a reliable change confidence level of 65%. Therefore we propose an RTclin cut-off value of 0 ms be used when screening an athlete with suspected concussion.

The test characteristics for RTclin reported here are similar to other recognised concussion assessment tools. For example, McCrea et al47 reported on the sensitivity and specificity of four commonly used assessment tools. Test sensitivities were highest when
testing occurred within 3 h of injury, with the Graded Symptom Checklist (GSC) being the most sensitive (89% sensitive, 100%
specific), followed by the Standardized Assessment of Concussion (SAC; 80% sensitive, 91% specific), and the Balance Error
Scoring System (BESS; 34% sensitive, 91% specific). The neuropsychological test battery administered 2 days following injury
had a sensitivity and specificity of 23% and 93%. At the same 2 day postinjury interval the sensitivity/specificity of the
GSC, SAC and BESS dropped substantially to 27%/100%, 22%/89% and 24%/91%, respectively. Overall a marked reduction in sensitivity
was observed for each test over the first week following injury. The authors also noted that combinations of tests had greater
sensitivity than individual tests. Broglio et al48 later reported on the sensitivity of five different concussion assessment tools.48 The most sensitive measure was the Immediate Post-concussion Assessment Cognitive Test (ImPACT; 79.2%), with its most sensitive
component being the symptom inventory (62.5%). The combined sensitivity of the remainder of the ImPACT battery was 62.5%,
with the memory and RT composite scores being the most sensitive individual tests (41.7% each). The overall sensitivity of
the HeadMinder Concussion Resolution Index was 78.6%, with individual test component sensitivities of 71.4% for simple and
complex RT and 50% for processing speed. A nine item concussion symptom scale was the next most sensitive tool (68%). As was
the case in McCrea's study, test batteries demonstrated greater sensitivity than any single component used in isolation. The
specificities of these tests were not reported. Other relevant studies in this regard are summarised in table 2.

Summary of relevant literature reporting on the test sensitivity and specificity of sport concussion assessment tools

In aggregate, the sensitivity and specificity of concussion assessment tools are greater when the test is conducted immediately
following injury, with a gradual decline in sensitivity over time. It also appears that test batteries with multiple components
are more sensitive than individual measures. With these considerations in mind, it is remarkable that RTclin, a single 5 min test that was performed up to 48 h after injury with equipment costing less than five US dollars, has comparable
test characteristics to other concussion assessment tools currently in use, including lengthier computerised neuropsychological
test batteries composed of multiple individual tests (eg, ImPACT and CRI), traditional paper and pencil neuropsychological
test batteries (composed of the HVLT, TMT, SDMT, DS, COWAT and Stroop), and costly balance assessment tools (eg, SOT). Further
research should evaluate the test characteristics of a concussion assessment battery composed entirely of clinical assessment
tools such as a symptom checklist, the SAC, RTclin and the BESS. Such a battery is likely to have improved sensitivity over its individual component measures, and could easily
be used by sports medicine providers in nearly any location at minimal expense.

The control athletes represent one of the study's strengths. These athletes were retested at similar time intervals as their
concussed team-mates and were typically receiving treatment for non-concussive injuries at the time of follow-up assessment.
This is important and suggests that the concussed athletes’ altered RTclin performance is more likely related to their concussions than to other in-season factors affecting the team that were not
measured or directly controlled for in our study design. In addition, the results are more compelling given prior work finding
RTclin to be valid, intrinsically motivating and functionally relevant.27–30,32 The study also has limitations that merit discussion. The majority of concussed athletes were male collegiate football players,
and so the generalisability of this study's findings to other populations is uncertain. Also, the athletes underwent after-injury
testing at a single time point within 48 h of concussion, and so no information regarding a postinjury ‘recovery curve’ is
available. Furthermore, since additional convergent and divergent measures were not simultaneously assessed during this study
and final athlete outcomes were not recorded, this study does not fully address the convergent, divergent or predictive validity
of RTclin for concussion. In addition, in some cases the follow-up tests on control athletes were not conducted on the exact same day
as the after-injury tests in their concussed team-mates. We do not suspect that these generally small differences in time
until follow-up testing have introduced significant bias, however their effect is not known. Finally, examiners were not blinded
to athletes’ concussion status during follow-up testing and so the introduction of measurement bias is possible. This is less
likely given the nature of the task, which occurs on a millisecond scale, and given that examiners were not aware of the athlete's
baseline performance.

In conclusion, the results suggest that RTclin performance is impaired following concussion, while in-season follow-up RTclin testing of non-concussed (but often injured) control athletes is associated with a small learning effect. Reliable change
calculations indicate that a critical RTclin change score of 0 ms (ie, interpreting any decline from baseline as indicative of impairment) is 75% sensitive and 68% specific
for concussion. These test characteristics compare favourably with most currently available concussion assessment tools, many
of which are associated with significantly more time, equipment, and expense than RTclin. Furthermore, RTclin may have even greater utility when used as part of a concussion assessment battery, in concert with a symptom assessment,
and clinical cognitive and balance assessments. RTclin, like any concussion assessment tool, cannot diagnose concussion in isolation and must be interpreted by a knowledgeable
sports medicine provider in the greater clinical context of the athlete being assessed. The results of this study support
the potential use of RTclin as part of a multifaceted clinical concussion assessment battery.

New findings

In this study, clinically measured reaction time (RT) distinguished between concussed and non-concussed athletes.

When any increase in clinically measured RT compared with the athlete's own baseline was considered abnormal (ie, a critical
cut-off value of 0 ms was used), the test sensitivity and specificity were 75% and 68%, respectively.

These findings support the use of clinically measured RT as part a multifaceted sports medicine concussion assessment battery.

Acknowledgments

The authors would like to thank all of the athletic trainers, team physicians, residents and students at Eastern Michigan
University, the University of Michigan, and Ann Arbor Skyline and Pioneer High Schools who assisted with data collection for
this study. Special thanks go to Steve Nordwall, Paul Schmidt, Rick Bancroft, Lenny Navitskis, Phil Johnson, Melissa Pohorence,
Bill Shinavier, Jennifer Garcia, Dave Cotner, Allyson Calhoun, Nate Santoni, Shailesh Reddy, Andy Schuldt and Tyler Ladue.
We would also like to thank all of the athletes and coaches whose teams participated in this study. Dr Eckner also thanks
the Rehabilitation Medicine Scientist Training Program for its support of his effort on this project through a K-12 career
development award.

Footnotes

Contributors Dr JTE contributed to the study conception and design, as well as the analysis and interpretation of data. He drafted the
article and provided final approval of the submitted manuscript. Dr JSK contributed to the study conception and design, as
well as the interpretation of data. He revised the article critically for important intellectual content and provided final
approval of the submitted manuscript. Dr SPB contributed to the analysis and interpretation of data. He revised the article
critically for important intellectual content and provided final approval of the submitted manuscript. Dr JKR contributed
to the study conception and design, as well as the analysis and interpretation of data. He revised the article critically
for important intellectual content and provided final approval of the submitted manuscript.

Funding Dr Eckner received a K-12 career development award from the Rehabilitation Medicine Scientist Training Program which supported
his effort on this project (grant number 5K12HD001097).

Competing interests An ICMJE conflict of interest form has been submitted by each author.