Baseline Neurocognitive Performance and Clearance for Athletes to Return to Contact

Neurocognitive evaluations have been an integral piece of concussion management for almost 3 decades.^1–3 In particular, computerized assessments have become widely used⁴ and are broadly known as computerized neurocognitive assessment devices or computerized neurocognitive tests (CNTs).⁵ The idea that CNTs provide incremental data that improve the health and safety of athletes has been controversial.^6,7 Proponents argue that CNTs offer advantages over symptom reports by supplying objective measurements of cognition after concussion; others question their true objectivity because of a reliance on individual effort.

The Immediate Post-Concussion Assessment and Cognitive Test (ImPACT; ImPACT Applications Inc, Pittsburgh, PA) is one of the most commonly used CNTs. This instrument uses 6 subtest modules to produce 4 composite scores: verbal memory, visual memory, visual motor speed (or processing speed), and reaction time. After a 1-week test-retest interval, Iverson et al⁸ reported Pearson correlations of 0.70, 0.67, 0.86, and 0.79 for the 4 composite scores, respectively. Using intraclass correlation coefficients (ICCs) to assess 1-year test-retest reliability, Elbin et al⁹ reported ICCs of 0.62, 0.70, 0.85, and 0.76, respectively. Schatz¹⁰ investigated ImPACT reliability over a 2-year span and found degradation of these values, reporting ICCs of 0.46, 0.65, 0.74, and 0.68 for verbal memory, visual memory, visual motor speed, and reaction time, respectively. In both baseline and postconcussion settings,^1,11,12 ImPACT has been validated against other neuropsychological measures and was shown to be approximately 90% sensitive and 70% specific in detecting cognitive dysfunction and diagnosing concussion,^3,13 even in the absence of reported symptoms.^13,14 However, other investigators have shown poor to moderate reliability,^6,15 low sensitivity,⁶ and a high (30%) false-positive rate.³

To determine if meaningful performance differences occurred postconcussion, ImPACT uses established reliable change criteria. Statistically, reliable change indices estimate the probability that a difference score was not simply the result of measurement error and also account for potential practice effects.^16,17 Clinically, reliable change criteria are used to increase the confidence that a decline in performance represents a true deficit rather than normal performance variability with repeat assessments. Applying the criteria determined by Iverson et al,⁸ ImPACT provides clinicians with clear indicators of whether or not an athlete is exhibiting performance deficits beyond the reliable change cutoff for each composite score. Yet because Iverson et al⁸ used a 1-week test-retest design, the calculated reliable change indices may not be as appropriate for an athlete whose baseline test was performed months or years before a concussion and subsequent testing. As such, the frequency with which baseline assessments should be performed is debated.^9,10

Similar to most neurocognitive tests, demographic and environmental factors can influence performance on ImPACT and other CNTs. Sex, age, test environment, mood disorder, and developmental disorders (eg, attention-deficit/hyperactivity disorder [ADHD], learning disability [LD], dyslexia) affect performance. Females outperform males on tests of verbal memory, whereas males outperform females on tests of visual memory.^18,19 Within the relatively limited age range studied, performance appeared to improve with age.^18–20 Testing individuals in a group setting can lead to worse test scores,^20–22 and the presence of mood or developmental disorders is also associated with worse performance.^23–25

Baseline scores are purportedly useful as a within-individual comparison standard when an athlete sustains a concussion; postinjury results are compared with baseline scores to determine whether and when the athlete is “back to normal.” However, baseline testing is potentially vulnerable to the effects of variable efforts²⁶ and purposeful poor performance, also known as malingering or “sandbagging.” Bailey et al²⁷ demonstrated that athletes with “suspect motivation” on traditional neuropsychological test measures at baseline showed disproportionate performance improvement 1 week after injury compared with “high-motivation” athletes at baseline. Anecdotal evidence suggests that athletes may attempt to generate a low baseline score in order to regain their preinjury baseline performance level more easily after a concussion and ultimately be cleared to return to play (RTP) more quickly.^28,29 To date, no investigators have examined differences in time to clearance for athletic participation based on mean baseline neurocognitive performance or the degree of reliable cognitive decline after concussion.

Culturally, this approach is associated with a de-emphasis on concussion concerns, consistent with previous research that identified high rates of concussion underreporting^30,31 as well as minimization of symptoms after concussion.³² Attributing worse-than-expected performance to suboptimal effort versus purposeful malingering is a challenge, and as such, many authors have had to rely on predetermined cutoff scores to identify potential instances without being able to definitively say the reason for low scores. From 6% to 11% of baseline scores may reflect poor effort or malingering.^10,22,33 Despite evidence that baseline assessments are susceptible to effort effects, other studies^34,35 showed that successfully sandbagging (ie, purposely performing poorly but still achieving a valid test score) on CNTs is difficult to accomplish. Considering the cultural belief that some athletes may attempt to obtain low baseline scores, it is worth investigating whether baseline scores are actually associated with elapsed time between concussion and RTP.

The purpose of our study was to evaluate mean baseline CNT performance between those with and those without postconcussion reliable decline and to examine mean differences in days missed after a concussion based on the number of domains in which an individual exhibited a reliable decline. We first assessed baseline performance differences between athletes who did and those who did not exhibit reliable declines postconcussion for each ImPACT composite score and hypothesized that athletes exhibiting a statistically reliable decline after concussion would have higher baseline scores than athletes who exhibited no change or an improvement after concussion, possibly reflecting the statistical properties of the test (ie, floor effects). We then investigated whether athletes exhibiting cognitive decline in a greater number of domains would, on average, lose more time from athletic activity and hypothesized that no such relationship would exist. Concussion management is multifaceted, and clinical decision making often reflects a variety of considerations at the individual level. If our hypotheses are correct, communicating such results in educational concussion-management programs has the potential to more accurately inform athletes about neurocognitive baseline testing. After a concussion, athletes may benefit from the understanding that broad cognitive deficits (ie, reliable declines in multiple domains) do not necessarily indicate they will have a longer recovery time.

METHODS

The University of Florida Concussion Databank (UFCD) contains concussion-related medical records for all consenting varsity athletes. With approval from the university institutional review board, we accessed the UFCD to obtain records for all University of Florida athletes who sustained a concussion between 2008 and 2015. For the present study, a concussion was defined as a brain injury diagnosed by a team physician or certified athletic trainer (or both). Our primary outcome variable, days missed, was defined as the number of days elapsed between the date of concussion and the date the athlete was cleared by a team physician to participate in contact activities. Both the concussion definition and the clearance guidelines for RTP were in accordance with the university's concussion-management protocol, which emulates the recommendations of the International Conferences on Concussion in Sport's consensus statements.^36,37 Our definition of clearance to RTP coincides with approval to begin step 5 of the graduated RTP protocol. This step represents the terminal step in the RTP process for noncollision-sport athletes and approval for a return to contact participation for collision-sport athletes. Although the definition of concussion and the clearance guidelines remained relatively consistent over the years encompassed within this analysis, we nonetheless evaluated the potential effect of changing treatment standards over time and found no correlation between year of injury and days missed (P = .471).

Computerized Neurocognitive Testing

University of Florida varsity athletes completed the online version of ImPACT at baseline and after a concussion under the supervision of a certified athletic trainer in accordance with the university's concussion-management protocol. Composite scores are obtained in 4 areas of cognitive functioning: verbal memory, visual memory, visual motor speed, and reaction time. For a detailed description of ImPACT composite scores and individual test modules, please refer to the ImPACT manual.³⁸

Postconcussion cognitive decline was defined as a statistically reliable performance decline relative to baseline. Reliable change statistics were based on previous psychometric analyses of ImPACT's test-retest reliability in high school and collegiate athletes (Table 1).⁸ Athletes were placed in 1 of 2 groups for each ImPACT composite score: (1) those who showed a reliable decline relative to baseline or (2) those who showed no change or a reliable improvement. Athletes were also grouped by the number of composite scores in which they exhibited a reliable decline (0, 1, 2, 3, or 4).

Table 1.  Criteria for Reliable Decline From Baseline for ImPACT Composite Scores (Derived From Iverson et al⁸)

View Fullscreen

Exclusion Criteria

Athletes were excluded if their baseline ImPACT performance was considered invalid based on the validity indicators described in the ImPACT manual.³⁸ These criteria are (1) >30 incorrect responses on the X's and O's test module, (2) impulse control composite score >30, (3) <69% correct on word memory learning, (4) <50% correct on design memory learning, or (5) <8 correct on the 3 letters test module.

Athletes were also excluded if any of the following information was missing from the UFCD: baseline ImPACT test scores before the date of injury, date of concussion, postconcussion ImPACT test scores, or date of clearance for RTP.

A total of 84 athletes met the inclusion criteria for this study (mean age at baseline = 18.9 ± 0.7 years, mean age when injured = 20.4 ± 1.3 years), and 29 were excluded. Sample characteristics are described in Table 2. Athletes in this study participated in the following sports: football (55), women's lacrosse (7), women's soccer (4), men's basketball (3), men's swimming and diving (3), women's basketball (3), women's volleyball (3), women's swimming and diving (2), women's track and field (2), men's tennis (1), and women's gymnastics (1). The first ImPACT administered after concussion was used for analysis (average time between concussion and ImPACT assessment = 2.9 ± 2.3 days, range = 0–10 days). Recently, our university began implementing annual baseline reassessments for all athletes. If an athlete had multiple baseline assessments on record, the baseline assessment closest to the date of injury was used (days between baseline and concussion = 545.2 ± 403.5, range = 5–1495 days).

Table 2.  Participant Characteristics

View Fullscreen

RESULTS

We evaluated associations between sample demographics and variables of interest using χ² tests. No associations were found between sex, concussion history, LD or ADHD, and presence or absence of reliable decline for any ImPACT composite score (all P values >.05).

Descriptive statistics for baseline composite scores, postconcussion composite scores, and percentage of athletes exhibiting reliable decline are shown in Table 3. For the paired-samples t tests, we conducted the Bonferroni correction (4 comparisons), resulting in a significance level of P < .0125. Overall, performance on the verbal memory (t₈₃ = 3.10, P = .003, d = 0.34, small-medium effect size) and visual motor speed (t₈₃ = 2.74, P = .008, d = 0.32, small-medium effect size) tests declined postconcussion. Reaction time performance decreased after concussion but not significantly after Bonferroni correction (t₈₃ = −2.51, P = .014, d = −0.36, small-medium effect size). Visual memory performance did not decline in this sample postconcussion (P = .135).

Table 3.  Participants Exhibiting Reliable Cognitive Decline for Each Composite Score on ImPACT

View Fullscreen

A substantial portion of our sample (40.5%) exhibited no reliable cognitive decline on any ImPACT composite score after concussion. Of the 84 athletes diagnosed with concussion, 20 (23.8%) exhibited a reliable decline on 1 ImPACT composite score, 16 (19.0%) exhibited reliable declines on 2 composite scores, 7 (8.3%) exhibited reliable declines on 3 composite scores, and 7 (8.3%) exhibited reliable declines on all 4 composite scores.

Baseline composite scores and days missed for athletes with or without postconcussion reliable decline are shown in Table 4. Athletes with a postconcussion reliable decline scored higher on baseline visual memory than athletes with no postconcussion change or a reliable improvement (t₈₂ = −2.35, P = .021, d = 0.65, medium-large effect size). No baseline performance differences were evident between athletes with postconcussion reliable decline and those with no postconcussion change or a reliable improvement for verbal memory (P = .420), visual motor speed (P = .112), or reaction time (P = .077). We did not apply the Bonferroni correction to the independent-samples t-test comparisons because the participants composing the reliable-decline and the no-change or improvement groups differed in each of the 4 domains.

Table 4.  Baseline ImPACT Performance and Days Missed Based on Postconcussion Cognitive Change Group

View Fullscreen

Days missed did not differ when comparing athletes who had postconcussion reliable decline with athletes who showed no change or a reliable improvement after concussion on verbal memory (P = .084), visual memory (P = .193), visual motor speed (P = .374), or reaction time (P = .356). Further, analysis of covariance demonstrated no main effect of the number of composite scores with a reliable decline on days missed (P = .530), and no significant covariates were identified (all P values >.05). In other words, athletes exhibiting reliable decline in most or all domains did not miss more days after injury than athletes exhibiting reliable decline in fewer or none of the cognitive domains.

DISCUSSION

We investigated differences in postconcussion outcomes based on mean baseline neurocognitive performance scores. Specifically, we examined whether groups of athletes who exhibited a reliable decline postconcussion on a given ImPACT composite score performed better at baseline than athletes who exhibited no change or improved. Athletes with a reliable decline after concussion had better baseline performance on just 1 composite score (visual memory). It is interesting that visual memory did not decrease in the total sample after concussion. This may indicate significant performance variability on visual memory tests. Previous evidence¹⁸ has shown sex differences in visual memory performance, with males performing better than females. Additionally, both verbal memory and visual memory tend to exhibit poorer score stability over time based on reliability data relative to ImPACT processing speed measures.^8,9

We also examined whether athletes with a reliable decline on a given ImPACT composite score took longer to achieve clearance for RTP. No statistical differences were noted in the number of days lost from athletic activity based on the presence or absence of a reliable decline on any ImPACT composite score. Additionally, athletes with a reliable decline in most or all of the ImPACT composite scores did not miss more time than athletes with a reliable decline in few or no composite scores. The descriptive data in Table 4 appear to show a clinically meaningful difference in days missed between those with and those without a reliable decline postconcussion, particularly for verbal memory and visual memory. However, because of the comparatively small number of participants in the reliable decline groups, the raw data are particularly susceptible to inflated means and standard deviations due to a few athletes who had protracted recoveries. Analyses using normalized outcome variables more accurately indicate the lack of a true statistical difference between these groups.

Only 60% of the athletes in our sample exhibited a statistically reliable deficit in performance on at least 1 ImPACT composite score based on the 80% reliable change index criteria of Iverson et al.⁸ This is in contrast to the findings of previous researchers^3,13 who cited sensitivity of more than 80% and may be partly attributable to the range of days between the injury and initial postconcussion ImPACT test in our sample. In an early study of change scores on ImPACT, version 2.0, Iverson et al⁸ observed that 75.6% of athletes exhibited reliable change relative to baseline on at least 1 composite score. However, unlike us, Iverson et al included the ImPACT postconcussion symptom scale composite score, which likely accounts for the discrepancy, given our exclusive focus on neurocognitive test performance. Regardless of methodologic differences, these discrepancies highlight the heterogeneity of postconcussion clinical presentation and neurocognitive performance, which are probably attributable to other factors (eg, injury severity, coincident symptoms, preinjury characteristics of the individual). For these reasons, a multifaceted approach to concussion management is always recommended.³⁷ The lack of association between cognitive decline and longer recovery time indicates that not all concussed athletes present with notable cognitive decline; their primary problem may be related to somatic-, vestibular-, or mood-based complaints.

The psychometric properties of ImPACT are 1 possible explanation for our findings. The 4 composite scores proposed by ImPACT have previously been questioned and may not represent a clear delineation of cognitive domains. Specifically, significant overlap in verbal and visual memory, as well as in visual motor speed and reaction time, provide evidence for a 2-factor structure: memory and speed composite scores.⁴⁰ Given that ImPACT does not contain any measure of simple reaction time but rather focuses on complex reaction time with decision-making components, the clinical utility of separating reaction time from visual motor speed on this CNT is perhaps most unclear. Evaluating recovery-time differences based on a modified factor structure is warranted. The limitations of ImPACT highlight the importance of clinicians' understanding the test's psychometric properties, particularly in the context of postconcussion performance interpretation.

Our results have implications for educating athletes on the purpose and clinical use of baseline testing, as well as managing recovery expectations after a concussion. We were unable to directly measure the possible effects of sandbagging or low effort on postconcussion performance or time missed from sport. Conceivably, athletes with low baseline scores in our sample included a mix of individuals extending sufficient effort but with poor cognitive abilities, individuals extending suboptimal effort, and possibly some who deliberately performed poorly in an attempt to sandbag. Steps must be taken to not only prevent sandbagging but also to promote maximum effort on baseline tests. If an athlete sustains a concussion and exhibits significant decline in multiple cognitive domains, clinicians can use these findings to help manage the athlete's expectations for recovery by sharing that individuals with broad cognitive deficits do not necessarily, on average, take longer to RTP than those with few or no areas of cognitive decline. We also recommend that clinicians, ideally in collaboration with a neuropsychologist trained in interpretation of their institution's neurocognitive assessment battery, carefully review the baseline test results of low-performing individuals to consider possible reasons for poor performance (eg, medical history indicates the presence of LD or ADHD) and the potential need to retest the athlete if suboptimal effort is suspected.

It is important to reiterate that we did not include athletes who were known or suspected to have purposely performed poorly on their baseline assessment. Previous authors^34,35 investigating athletes' ability to successfully sandbag found this was difficult to accomplish without detection by invalidity indicators. Our study likely included a broad range of athlete motivation levels. Future researchers may be able to investigate this phenomenon more directly if they can identify and report outcomes in athletes who either admitted to or showed evidence of sandbagging their baseline test and then later sustained concussions.

This study is limited by its retrospective design. We could not control when the first ImPACT assessment was administered after the incident concussion. Clinicians often differ in how they choose to use CNTs. Some use them early in the postinjury period for diagnostic assistance, whereas others prefer to use them later in the recovery process as an indicator for readiness to RTP. Indeed, the first assessment occurred over a wide range of days (0 to 10 days); future authors should prospectively investigate the relationship between CNT performance and days missed after concussion using multiple defined assessment points.

Also, this sample was disproportionately represented by males and football players and was limited to National Collegiate Athletic Association Division 1 collegiate athletes. Therefore, the findings may not generalize to both sexes, different sports, or all levels of participation. Although the nonnormalized descriptive statistic of days missed seemed to indicate clinically meaningful differences based on the presence or absence of reliable cognitive decline, further analyses with a larger and more diverse sample would contribute to our understanding of this relationship. Lastly, acute factors that may predict injury severity, such as loss of consciousness and posttraumatic amnesia, were not readily available for all patients in our study and may have contributed to recovery variability.

Despite these limitations, the present study may more closely represent actual clinical practice, in which variable assessment points are often used. In addition, most previous investigations have focused on the sensitivity of a CNT to detect injury or indicate the time needed for athletes to return to baseline performance; however, we focused on the clinically meaningful outcome of time missed from athletic activity.