Invalid Performance and the ImPACT in National Collegiate Athletic Association Division I Football Players

Sport-related concussions have been a growing topic of interest in the popular media and clinical neuropsychology.¹ Concussions, often termed mild traumatic brain injuries, are defined as traumatically induced, typically reversible impairments of neurologic function. Researchers^2–5 have estimated the prevalence of US sport-related concussions at about 300 000 per year. This figure is likely an underestimate because many concussions are not reported.⁶ Given its popularity and perceived risk of concussions, football is the most frequently examined sport, and an estimated 5% to 9% of collegiate players sustain concussions each year.^2,4,7 Across various college sports, concussions compose 5.8% to 6.2% of all reported injuries.^3,7

Several computerized cognitive test batteries have been developed to assist in the diagnosis of sport-related concussions.^8–10 Nearly 95% of athletic trainers in a recent survey reported conducting baseline cognitive testing of some form.¹¹ However, only 51.9% of those athletic trainers who administered computerized baseline testing examined the tests for validity concerns.¹¹ Without valid baseline testing, determining whether postinjury testing shows a meaningful change from previous levels of functioning can be challenging.¹² For example, Hunt et al¹³ showed 11% of high school athletes given a brief neuropsychological test battery exhibited poor effort on testing and athletes with invalid protocols had lower scores on several tests within the battery.

A common computerized neuropsychological test is the Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT), which is used not only by the National Football League and National Hockey League but by several US Olympic teams and many universities throughout the United States and Canada.¹² The ImPACT manual¹² provides indices to detect invalid responding during testing, but no one has examined the prevalence of invalid test profiles within a sample of student-athletes tested under typical conditions. Data from student volunteers suggest that 25.7% exhibit suboptimal effort on testing.¹⁴ Therefore, the purpose of our study was to determine the prevalence of invalid responding in student-athletes by examining preseason ImPACT testing in football players across a 3-year period. We hypothesized that poor effort would be common within this sample because researchers^13,14 have shown that 11% of high school students and more than 25% of student volunteers exhibited suboptimal effort on neuropsychological testing. Furthermore, athletes may be motivated to intentionally suppress their baseline performances to limit detection of concussions during the season and to be allowed to continue to participate.

METHODS

Measures

The ImPACT is a computerized neuropsychological test battery aimed at assessing multiple areas of cognitive functioning, including attention, concentration, memory, processing speed, reaction time, and concussion-related symptom reporting. Upon completion, 5 composite scores, including verbal memory, visual memory, visual motor speed, reaction time, and impulse control, and a total symptom score are generated.¹² Reliability studies have produced mixed results for these indices, with intraclass correlation coefficient estimates ranging from 0.15 to 0.61 in a test-retest study¹⁴ in which researchers examined baseline to 45 days and 45 days to 50 days postbaseline. In another test-retest study,¹⁵ investigators found intraclass correlation coefficient estimates ranging from 0.42 to 0.74 over 2 years. When compared with traditional neuropsychological measures, processing speed and reaction time measures have moderate correlations with similar traditional neuropsychological measures.^16,17 Schatz et al¹⁸ examined a score that consisted of the visual memory, processing speed, and impulse control composite scores and found the sensitivity of ImPACT was 81.9% and the specificity was 89.4% for identifying individuals with concussions.

The ImPACT clinical interpretation manual¹² describes 2 types of invalid protocols: invalid profiles and “sandbagging” profiles. Invalid profiles meet predetermined criteria based primarily on composite scores that indicate an athlete has not performed to his or her true level, so the results are inaccurate (Table 1). These profiles may be invalid because the participant did not read or understand directions, had attention or learning problems, was fatigued, was distracted, or had left-right confusion. Several ranges of scores are given for invalid assessments, including low percentages for learning on memory tests, high incorrect or low correct scores for 2 subtests, and high impulse control composite scores. More specific ranges also are given for sandbagging scores, a specific subtype of invalid profile that denotes feigning weakness. Sandbagging indicates an athlete intentionally is suppressing his or her performance with the likely intent to hide any impairment when comparing baseline scores with postconcussion assessment. Sandbagging scores are based on low verbal and visual memory composite scores and slow reaction time composite scores. According to the ImPACT manual,¹² reaction time composite scores in this range generally fall below the fifth percentile (Table 1). Given that the ImPACT manual does not offer descriptive statistics or reliability data on these indices, we do not know how accurate these indices are in truly classifying protocols as invalid or feigned. These indices have never been compared with other formal measures of effort or malingering. Furthermore, no empirical literature is available to support that individuals denoted as sandbagging in fact were actively feigning cognitive deficits.

Table 1.  Invalid Responses for Each Test Measure (N = 269)

View Fullscreen

RESULTS

Factors Associated With Suboptimal Effort on ImPACT Testing

We examined a series of demographic and medical variables to identify factors that may increase the likelihood of suboptimal effort on ImPACT testing (Table 2). Individuals with histories of special education were more likely to be identified as producing invalid responses during testing (8.0% [n = 6] versus 2.1% [n = 4]; χ² = 0.02, P = .02). No such differences emerged for factors such as years of education or history of hyperactivity. To clarify these findings, we conducted logistic regression with all demographic and medical variables (age, education, number of previous concussions, total symptom score, history of hyperactivity, history of repeating a grade, history of special education, and diagnosis of a learning disorder) to determine whether they could predict which athletes were more likely to exhibit invalid responding during testing. The dependent variable looked at individuals with any invalid index compared with those with valid profiles (Table 3). The overall model was different (Nagelkerke R² = 0.32, P = .02); however, age appeared to be driving the model χ² (χ² = 0.01, P = .02), and no other variables emerged as predictors of valid responding (P > .05 for all). Individuals with valid protocols (age = 19.91 ± 1.41 years) on average were older than those with invalid protocols (age = 19.59 ± 1.36 years).

Table 2.  Patient Characteristics Based on Invalid Responses

View Fullscreen

Table 3.  Logistic Regression Analysis Examining the Predictive Validity of the Descriptive Variables Among 262 ImPACT Protocols (Full Model)

View Fullscreen

DISCUSSION

Our results suggest that invalid and sandbagging performances are common on the ImPACT test because more than 25% of Division I football players at our institution produced a baseline score that suggested suboptimal effort according to the ImPACT manual. Although a brief informational session on concussion risks did not improve rates of valid responding, instructing athletes to complete the ImPACT a second time did appear to improve them. This improvement does not necessarily indicate that the individuals originally were feigning difficulties, because situational or longstanding factors may have affected these scores. Several aspects of these findings warrant brief discussion.

In our study, more than one-fourth of college athletes produced scores that were invalid on computerized cognitive testing, even when provided with information about the potential risks of concussion. Poor effort may complicate interpretation of postconcussion testing and place athletes at risk for subsequent injury because distinguishing poor test performance due to mild traumatic brain injury from suboptimal effort is difficult.^19,20 The ImPACT manual¹² provides guidelines for detecting invalid and sandbagging performances, and when an ImPACT report is printed, a notice is included if the individual produces a problematic profile. However, Covassin et al¹¹ found that just over half of computerized test examiners used the validity scales at baseline. Future studies are needed to determine the best practices for using validity indices from the ImPACT and other computerized test batteries because such work may help to decrease concerns regarding the ability of baseline testing to reduce concussion risk in athletes.²¹ Similarly, additional work is needed to more clearly delineate the validity indices on the ImPACT test battery. In its current form, the ImPACT manual does not offer a clear rationale for the formulation of the validity indices, and no one has directly examined their psychometric properties, including sensitivity and specificity. We do not know how sandbagging indices differ from the other invalid indices and the degree to which they provide distinct and useful data to test administrators.

As noted, 27.9% of ImPACT assessments indicated invalid responding in our study, which is a pattern generally consistent with rates in student volunteers (25.7%)¹⁴ but considerably higher than found in high school athletes (11%).¹³ However, this pattern is not surprising because Hunt et al¹³ used well-validated, individually administered tests, and rates of poor effort are typically lower on individually versus group-administered tests.²² Unfortunately, individually administering paper-and-pencil tests to athletes often is not practical for many programs. Even our midsized university conducts more than 200 preseason evaluations each year, and the extra resources for such evaluations may not be readily available.

Such findings suggest a strong need to identify strategies to improve the validity of computerized concussion testing. As mentioned, inspecting scores after testing would be appropriate to improve validity. A brief information session about concussion safety did not affect the prevalence of poor effort in our study. Researchers should examine whether other educational interventions may be more effective. However, such approaches may be difficult in student-athletes because knowledge of risk alone often is insufficient to produce behavioral change.^23,24 A more effective approach may involve personally requesting student-athletes to undergo retesting, as 7 of 8 athletes exhibiting invalid responding improved their test performance on repeat testing. Larger studies are needed to replicate this finding and examine whether the threat of repeat testing reduces the prevalence of suboptimal effort at the group level over time.

When combined with past work, our results suggest a possible approach to improve the validity of baseline ImPACT test performance. First, individuals administering the ImPACT test are encouraged to follow the recommendations of Lezak et al²⁵ to obtain valid testing data, including providing a quiet testing space, removing distractions (eg, cell phones), and informing athletes about the importance of providing optimal effort. Other improvements to the testing environment include having trained personnel monitor testing; reducing group numbers; and taking note of athlete characteristics that may influence test scores, such as illness, sleep, and stress. All test performances should be screened immediately upon completion to identify the possible presence of an invalid or sandbagging profile. Athletes should be informed of their questionable test performances, and they should repeat testing at least 24 to 48 hours later. This delay is encouraged to alleviate potential fatigue.²⁶ Given that the potential practice effects are unknown, investigators should determine the appropriate interval between testing sessions to optimize the clinical utility of these tests. If repeat testing indicates invalid or sandbagging profiles, careful discussion with the athlete may reveal factors, such as illness, sleeping problems, attention or learning problems, or psychological distress, that may have contributed to his or her performance.^12,27 For example, student-athletes with histories of special education were more likely to be identified as demonstrating suboptimal effort on testing in our study. Therefore, invalid profiles actually may be valid results, which are lower than the norm because of the legitimately weak cognitive skills of the athlete. Depending on the outcome of this conversation, referring the athlete for a more thorough neuropsychological evaluation may be beneficial because he or she may have an undiagnosed attention or learning disability.¹² Empirical research is needed to examine whether this approach can reliably produce a higher proportion of valid scores.

CONCLUSIONS

More than 25% of college football players' baseline cognitive testing suggested suboptimal effort according to the ImPACT manual. Providing information about concussion risks and the importance of cognitive testing did not reduce the prevalence of poor effort, but instructing athletes to repeat testing did. Further research is needed to identify effective and efficient strategies that reduce the prevalence of invalid performances on the ImPACT.