Between-Seasons Test-Retest Reliability of Clinically Measured Reaction Time in National Collegiate Athletic Association Division I Athletes

Concussion is a common and potentially serious injury that frequently results from sport participation. Recent management guidelines12 emphasize a multifaceted approach to concussion assessment with the goal of determining that an athlete has fully recovered before returning to play. Full recovery from a concussion includes subjective symptom resolution and complete normalization of the physical examination, including postural stability and cognitive assessment. Over the past 2 decades, the use of computerized concussion assessment batteries such as the Immediate Post-Concussion Assessment and Cognitive Test (ImPACT; ImPACT Applications, Inc, Pittsburgh, PA),3 CogState Sport (CogState Ltd, Melbourne, Australia),4 and the Automated Neuropsychological Assessment Metrics (ANAM; Defense and Veterans Brain Injury Center, Washington, DC)5 has grown substantially to assist sports medicine professionals in assessing a concussed athlete's cognitive recovery. One measure that is included in all widely used computerized concussion assessment batteries is reaction time.

Impaired reaction time after sport-related concussion has been demonstrated repeatedly.6–10 Reaction time measures provide one of the most sensitive indices of cognitive changes after concussion in both athletic and general head injury populations.1112 Decreased speed of information processing is thought to account for the cognitive performance deficits seen after concussion.11 Impaired reaction time after concussion parallels the persistence of postconcussive symptoms79 and may persist beyond resolution of self-reported symptoms and clinical findings.8 Reaction time assessment can add sensitivity to the sports medicine practitioner's concussion assessment battery. However, the measurement of reaction time relies on specialized computer programs, which limits its accessibility to many athletes, especially younger athletes, who are traditionally underserved by the sports medicine community as compared with athletes at the collegiate and professional levels. This point is especially concerning given that people aged 5–18 years account for 65% of all sport-related concussions.13

To provide clinicians with a reaction time measure that does not rely on computers, we14 developed a simple clinical method for measuring reaction time (RT_clin). Pilot reliability and validity studies14 in a small sample of healthy adult volunteers demonstrated excellent test-retest and intertester reliabilities for this technique and conformity to known reaction time characteristics, such as slowing with age and dual task, as well as correlation with a computerized reaction time measure. We15 have also demonstrated a significant positive correlation between RT_clin and the simple reaction time component of the CogState Sport computerized neuropsychological test battery in a sample of collegiate football players during preparticipation physical examinations. Furthermore, we16 have shown RT_clin to strongly correlate with a test participant's ability to protect the head using the hands during a laboratory task designed to simulate a sport-specific protective response. In ongoing research, we17 are investigating the effect of concussion on RT_clin in collegiate athletes using a postinjury comparison with baseline design.

It is critical to determine the stability of any measure intended to be used serially, as when comparing postinjury results with a preseason baseline. Only a small body of literature describes the test-retest reliability over various time intervals of currently available computerized sport concussion assessment batteries, each including a computerized measure of reaction time. Generally speaking, such studies have demonstrated the greatest stability over short test-retest intervals, on the order of 1–2 weeks, and decreasing reliability over intervals up to 2 months. Season-to-season comparisons are rare, despite these being common retest intervals in practice.

In pilot work, we14 reported a test-retest intraclass correlation coefficient (ICC) of 0.860 (P = .004) for RT_clin in a sample of 10 healthy adult volunteers tested by a single examiner on 2 occasions within 1 month. However, little can be inferred from these data with respect to the test's long-term stability in athletes given the small sample size, the lack of inclusion of athletes, and the short between-tests time interval. Therefore, the purpose of our current study was to measure the 1-year test-retest reliability of RT_clin in a larger sample of National Collegiate Athletic Association Division I athletes. To accomplish this, we compared year 1 and year 2 mean RT_clin values in athletes who completed preseason RT_clin testing over 2 consecutive seasons. This testing was performed as part of an ongoing concussion monitoring program. A secondary goal of the study was to compare the stability of RT_clin over seasons with that of a concurrently administered computerized measure of reaction time (RT_comp), namely the simple reaction time component of the CogState Sport test battery.

METHODS

Measurement of RT_clin

Clinical reaction time is determined using a simple, manual visuomotor task: the time needed to catch a suspended vertical shaft by hand closure. The device is equipped with a weighted spacer at the lower end to ensure near verticality and standardize finger closure distance. Testing took place in an isolated room when one was available and otherwise in the corner of a larger room where the athletes waited as they moved between components of the preparticipation examinations. The RT_clin test protocol used has been previously described.1415 In brief, the athlete sat with the forearm resting on a horizontal desk or table surface with the hand positioned at the edge of the surface. The athlete held the hand sufficiently open to fit around, but not touch, the weighted disk portion of the clinical reaction time apparatus. The examiner suspended the apparatus and released it after randomly determined time delays between 2 and 5 seconds so as to minimize the athlete's ability to anticipate release of the device. Upon release, the athlete caught the device as rapidly as possible by hand closure (Figure). Clinical reaction time was calculated from the fall distance of the device using the formula for a body falling under the influence of gravity (d = 1/2gt²), with fall distance measured from the most superior aspect of the athlete's hand after catching the device. Anticipatory grasps and “drop” trials were excluded, as previously described.15 Each athlete performed 2 practice trials, immediately followed by 8 data acquisition trials.

View Figure

• Download as Powerpoint
• Download Figure

RESULTS

Of 251 student-athletes who participated in the study, 102 athletes completed baseline RT_clin test sessions during 2 consecutive seasons and were included in the data analysis. Of these, 78 played football, 14 wrestled, and 10 played women's soccer. Sixty-two of these athletes had 2 corresponding valid RT_comp data points and were included in the RT_comp analysis. Of the 40 athletes excluded from RT_comp analysis, 27 had at least 1 invalid CogState Sport session and 13 had missing CogState Sport data from athlete identifier coding errors.

The overall mean RT_clin and RT_comp values and their SDs, as well as ICCs and SEMs for RT_clin and RT_comp, are presented in Table 1. The RT_clin had a higher overall test-retest ICC than did RT_comp, and the SEM was lower for RT_clin than for RT_comp. In athletes with valid CogState Sport data for comparison, mean RT_clin was 50 milliseconds shorter than mean RT_comp in year 1 (t = −12.01, P < .001) and 62 milliseconds shorter in year 2 (t = −12.02, P < .001). A learning effect occurred in RT_clin (year 1 to year 2: 11 milliseconds, t = 4.66, P < .001), whereas RT_comp did not differ between the years (t = −0.16, P = .875). The RT_clin did not differ between athletes with and without valid RT_comp data for comparison in year 1 (205 ± 22 milliseconds versus 209 ± 25 milliseconds, respectively; t = −0.894, P = .373) or year 2 (194 ± 23 milliseconds versus 198 ± 25 milliseconds, respectively; t = −0.834, P = .406). Furthermore, the ICCs for RT_clin in athletes with and those without 2 valid CogState Sport computerized test sessions (ICC = 0.548 and ICC = 0.746, respectively) both fall within the overall 95% confidence interval for RT_clin (0.422, 0.775).

Table 1. Overall Group Results for Clinical and Computerized Measures of Reaction Time

View Fullscreen

DISCUSSION

The results of this study demonstrate that RT_clin has sufficient long-term stability, as measured by its test-retest reliability over successive seasons, to be used serially as a concussion assessment tool for preseason baseline to postinjury comparisons. The 1-year reliability estimate for RT_clin (ICC = 0.645) compares favorably with that of concurrently measured RT_comp. Although the ICC for RT_clin obtained in this study is somewhat lower than ICCs in the literature for computerized reaction time measures over retest intervals of 1 hour to 2 weeks,31112 it is comparable with or greater than published values for time intervals ranging from 45 days to 2 years.2123 Table 2 summarizes literature-based reliability estimates for the reaction time components of ImPACT, CogState Sport, Concussion Resolution Index (Head-Minder, Inc, New York, NY), and Concussion Sentinel (Cog-State Ltd). Overall, the 1-year stability of RT_clin falls within the range of values reported for computerized reaction time measures that are currently in widespread use.

Table 2. Summary of Studies Reporting Test-Retest Reliability for Computerized Reaction Time Measures

View Fullscreen

The level of test-retest agreement thought to be acceptable for clinical use varies among authors. The 1-year RT_clin ICC falls above the minimum acceptable ICC value of 0.60 proposed by Anastasi24 but below the 0.90 ICC value cited by Randolph et al25 as desirable for making decisions about individual change in the context of sport-related concussion. In reality, it is rare for a clinical test to achieve test-retest reliability greater than 0.90, and none of the currently available and widely used computerized concussion batteries have yielded ICCs for reaction time or any other test measure that approach this level for re-test intervals longer than 1 hour. Although reliability estimates of ICC > 0.90 would be ideal, this appears to be an unrealistic benchmark for a test assessing something as complex as the brain's processing speed over prolonged time periods. Tests with lower reliability indices, as demonstrated for RT_clin and the numerous computerized reaction time measures detailed here, can still be of clinical value.

An important contributor to a test's reliability is the learning effect associated with repeated administration of the test. Even though the pilot RT_clin reliability and validity study did not suggest a learning effect in nonathletes,14 a learning effect over 8 RT_clin trials appeared to be present during our initial baseline assessment15 of RT_clin in 94 collegiate football players. The present study further suggests the possibility of a learning effect in athletes by demonstrating an average RT_clin decrease of 11 milliseconds from year 1 to year 2. The test protocol for determining RT_clin in athletes has been consistent across studies, with participants given 2 practice trials followed by 8 data acquisition trials. Investigation into the effect of more practice trials before data collection is warranted.

It is noteworthy that almost one-third of the athletes who participated in this study had one or more invalid CogState Sport test sessions. Although a variety of reasons for an invalid computerized test session exist, poor effort or motivation on the part of the athlete is probably one of the most common. The finding that RT_clin did not differ between athletes with and without valid RT_comp data for comparison parallels our prior findings15 and suggests that RT_clin may be immune to this concern. Furthermore, RT_clin may have actually been more stable across seasons in the athletes without valid RT_comp data for comparison, that is, primarily athletes excluded from RT_comp analysis due to invalid CogState Sport sessions. A possible explanation is that RT_clin may be more intrinsically motivating than computerized reaction time tasks. This could be because of several factors, including the more physical nature of the RT_clin task and the direct one-on-one interaction inherent in RT_clin testing that is absent from computerized test batteries. Less than full effort and motivation on cognitive test performance has a potent influence on outcome, as demonstrated by Green et al,26 who showed that suboptimal effort suppressed overall test battery performance 4.5 times more than did moderate to severe brain injury. Ensuring optimal effort is critical when preseason baseline tests are the basis for comparison with postinjury testing, when an athlete is often highly motivated to perform well so as to return to play. Therefore, an intrinsically motivating test is likely to provide higher-quality baseline test results. Additional study is necessary to specifically address the role of motivation in baseline reaction time testing.

The strength of the conclusions in this study must be tempered by the limitations. We investigated only Division I athletes, 76% (78/102) of whom were football players and 90% (92/102) of whom were male. Therefore, the results are applicable primarily to this population and may not generalize to female athletes or athletes participating at high school or youth levels. We would not expect test-retest reliability to differ in these populations, but additional study in females and younger populations is warranted. This is especially true given that the simplicity and anticipated low cost of RT_clin may make it most valuable in younger athletes who do not have access to computerized neuropsychological test batteries. To our knowledge, Erlanger et al11 provided the only test-retest reliability estimates for computerized concussion batteries that allowed comparison of athletes across age groups. The reliability estimates reported do not differ greatly between high school students and collegiate or adult club athletes on processing speed (0.79 and 0.90, respectively), simple reaction time (0.72 and 0.73, respectively), or complex reaction time (0.65 and 0.72, respectively). In addition, the results may not apply to the shorter time periods that more closely approximate the typical timeframe seen in sport-related concussion assessment and management.21 An additional study limitation is the lack of counterbalancing in the order of RT_clin and RT_comp testing. This lack of control was pragmatic because of the logistics of testing large numbers of athletes during their preparticipation physical examinations. We do not hypothesize any order effect, but the true effect of test order is unknown. A final limitation of this study is that the RT_comp data used for comparison with RT_clin were not of optimal quality. All athletes completed only a single CogState Sport test session each season, and 40 athletes had at least 1 missing or invalid CogState Sport test session. With regard to the 13 athletes with missing RT_comp data, a systematic problem in the way athlete identifiers were coded within the CogState system in the first year of testing, which was subsequently fixed, is to blame. Regarding the large number of invalid CogState Sport test sessions with unusable data, authors of future prospective studies should consider a double-baseline CogState Sport test protocol to reduce the learning effect associated with the test and improve data quality.27 Although it would not affect the primary outcome variable of this study (RT_clin stability), this change in protocol may allow a better comparison of RT_clin with RT_comp.

In summary, the test-retest reliability of RT_clin over consecutive seasons compares favorably with computerized reaction time measures. This suggests that it is a stable measure across seasons and, taken in the context of our previous work,14–17 supports its potential use as part of the sports medicine practitioner's multifaceted concussion assessment battery. We caution that impaired simple reaction time is only one of many typical signs indicating that an athlete has sustained a concussion. Therefore, it must be interpreted within the greater clinical context of the concussed athlete. Furthermore, in athletes who have access to computerized neuropsychological testing, RT_clin should not be considered a replacement for computerized tests, which measure multiple indices of concussion in addition to simple reaction time. However, the simplicity and low cost of RT_clin could facilitate its use in youth athletes and others who do not have access to computerized concussion assessment programs. In athletes who do have access to computerized testing, RT_clin may serve a complementary role as a true sideline tool in the initial concussion diagnosis, when use of computerized test batteries is impractical. Further study is warranted to examine the influence of motivation on baseline reaction time assessment, evaluate the use of RT_clin in younger populations, and prospectively investigate the effect of concussion on RT_clin. Controlled research involving concussed athletes will also need to define a clinically meaningful change in RT_clin from baseline on which management decisions can be based.