A Comparative Meta-Analysis of the Effects of Concussion on a Computerized Neurocognitive Test and Self-Reported Symptoms

Data Extraction and Coding

Two raters (K.S., D.P.) completed a thorough review of the titles and abstracts of the studies. They also independently reviewed the full text of each study and extracted the data using an electronic spreadsheet. Disagreement between the raters regarding the retrieved data was resolved by consensus. The sample size, demographic characteristics of participants, time between concussion and postconcussion ImPACT testing, ImPACT composite scores, and PCSS scores were retrieved in addition to the scores for applicable time points (baseline, within 1 week of concussion, and between 1 and 3 weeks after concussion). If ImPACT was completed more than once within the periods specified for this study (within 1 week and between 1 and 3 weeks), the scores of the first testing session within a given interval were chosen for this meta-analysis. For studies in which the authors reported ImPACT scores for more than 1 sample, we assumed that these samples were independent and reported them separately.

Assessment of Reporting and Methodologic Quality

We used the Strengthening the Reporting of Observational studies in Epidemiology (STROBE) statement to assess the reporting quality of the reviewed studies.²⁰ The STROBE statement evaluates 22 aspects related to the reporting of observational studies. Each aspect was assigned a numeric value of 1 if it was explicitly described and presented in a study and 0 if it was inadequately described or absent in a study. As such, a total score out of 22 was calculated to reflect the overall reporting quality of each study; a higher score indicated better reporting quality. We did not use reporting-quality scores as moderators in the pooled analysis because of the potential to confound the findings.

Meta-Analysis Process

Search Yield

The search yielded 5943 studies. The raters examined the study abstract and identified 26 studies for full-text review. After reviewing the 26 studies and reapplying the same inclusion criteria, we excluded 9 studies. Reasons for exclusion were the absence of the total PCSS score (n = 5),^30–34 use of ImPACT version 1 (n = 2),^35,36 absence of baseline data (n = 1),³⁷ and lack of details regarding testing time points (n = 1).¹³ As such, 17 studies consisting of 29 independent samples were included (Figure).

View Figure

• Download as Powerpoint
• Download Figure

Characteristics of Included Studies and Reporting Quality

A total of 1777 patients (1250 males, 527 females) with concussion were included in this meta-analysis. Individual studies included participants from 13 to 33 years old representing clinical populations that routinely complete ImPACT during the clinical management of concussion. Participants in the reviewed studies included middle and high school–aged children, college-aged adults, and professional athletes. The sample sizes in the reviewed studies varied from 15 to 222 participants. Twenty-nine independent samples were included: 26 samples were included in the within-1-week analysis, and 10 samples were included in the 1- to 3-week analysis. The average (standard deviation) time points considered for the within-1-week analysis and the 1- to 3-week analysis were 2.2 (1.4) days and 9.7 (2.6) days, respectively. In 7 samples, ImPACT was serially administered at both time points; therefore, the designated ImPACT scores were considered for the appropriate analysis (Table 1). The reporting quality was generally moderate to high, and STROBE scores ranged from 17 to 21 (Table 1).

Table 1.  Characteristics of Included Studies

View Fullscreen

Heterogeneity

The within-1-week analysis demonstrated heterogeneity for all outcomes (P ≤ .007), with low heterogeneity for processing speed (I² = 45.37) and medium heterogeneity for all other ImPACT and PCSS scores (I² = 73.42), further indicating the level of unexplained variance across studies (Table 2). For the 1- to 3-week analyses, we observed heterogeneity for verbal memory, reaction time, and PCSS score, with the unexplained between-studies variance ranging from 54.53% (PCSS) to 73.97% (verbal memory; Table 3).

Table 2.  Effect Sizes of Change (Hedges g) and Heterogeneity Estimations in Immediate Post-Concussion Assessment and Cognitive Testing^a and Postconcussion Symptom Scale Scores for Independent Samples Within 1 Week Postconcussion (k^b = 26)

View Fullscreen

Table 3.  Effect Sizes of Change (Hedges g) and Heterogeneity Estimations in Immediate Postconcussion Assessment and Cognitive Testing^a and Postconcussion Symptom Scale Scores for Independent Samples Between 1 and 3 Weeks Postconcussion (k^b = 10)

View Fullscreen

Effect Size

All ImPACT and PCSS scores demonstrated ESs of change, which were adjusted for time from injury to follow-up correlation, that were different for studies when the postinjury follow-up measurement was within 1 week (Table 2; Appendix 1A through E). The greatest change was observed in PCSS score (Hedges g = −0.81), and the smallest change was seen in processing speed (Hedges g = −0.43). Comparing the ES CIs, the overall ES of change was greater for the PCSS (Hedges g = −0.81, 95% CI = −0.91, −0.71) than for reaction time (Hedges g = −0.58, 95% CI = −0.68, −0.47) and processing speed (Hedges g = −0.43, 95% CI = −0.52, −0.35). Similarly, the overall ES of change was greater for visual memory (Hedges g = −0.64, 95% CI = −0.74, −0.54) and verbal memory (Hedges g = −0.67, 95% CI = −0.77, −0.58) than for processing speed.

A small but different ES was observed for all ImPACT and PCSS scores measured between 1 and 3 weeks postconcussion, with the greatest effect in the PCSS (Hedges g = −0.38) and the smallest effect in visual memory (Hedges g = −0.25; Table 3; Appendix 2A through E).

Meta-Regression of Unexplained Between-Studies Variance

Across the 26 samples with outcomes measured within 1 week, meta-regression identified time since concussion as a univariate moderator for all ImPACT scores. As time since concussion increased over the 7 days, a smaller decline in cognitive scores was observed. A larger postinjury PCSS score was associated with a larger cognitive decline ES for verbal and visual memory (Table 4). Time postinjury and postinjury PCSS score influenced processing speed ES, but only time postinjury was different from both moderators in the model. The greatest amount of between-studies variance was explained by both time postinjury and follow-up PCSS score for verbal memory (81%). Time postinjury was a moderator for reaction time, which explained the 5% of the between-studies variance (Table 4). Preinjury PCSS score, percentage of females, and concussion history were not associated with ES within 1 week postconcussion. Time postinjury did not explain the degree of between-studies variance in PCSS scores.

Table 4.  Meta-Regression Results for Time to Measurement and Follow-up Postconcussion Symptom Scale Scores as Moderators of 1-Week Effect Size of Change in Immediate Postconcussion Assessment and Cognitive Testing^a Scores (k^b = 26)

View Fullscreen

Meta-regression results indicated that preinjury PCSS score moderated the reaction time ES in studies with outcomes measured between 1 and 3 weeks postconcussion (β = 0.09, z = 2.06, P = .04), whereby greater preinjury PCSS score was associated with a smaller decline in postinjury reaction time. A model using the preinjury PCSS score explained 37% of the between-studies variance in reaction time change preinjury to postinjury. Time postinjury was a moderator of PCSS score ES (β = 0.06, z = 3.78, P < .001), with the model explaining 100% of the between-studies variance; the PCSS score ES from preinjury to postinjury (between 1 and 3 weeks) decreased as the postinjury measurement time increased. No tested covariates moderated the ES for verbal memory. Percentage of females and concussion history were not moderating influences on the ES of change in any outcome.

For the meta-regression of 1 to 3 weeks postinjury, the post hoc analysis showed that the power for the time regression coefficient was 24% for verbal memory and 6% for reaction time.

Publication Bias

For the samples with outcomes measured within 1 week postconcussion, funnel-plot inspection supplied evidence for possible publication bias for all ImPACT and PCSS scores. The Egger regression test indicated publication bias for reaction time (β = 1.83, t₂₄ = 1.85, P = .04) and PCSS score (β = 3.31, t₂₄ = 3.60, P = .001; Table 5). For ImPACT reaction time, the trim-and-fill method suggested that 6 studies were missing and the point estimate for Hedges g, adjusted for publication bias, was −0.67 (95% CI = −0.77, −0.56). For PCSS score, the trim-and-fill method suggested that 10 studies were missing and the point estimate for Hedges g, adjusted for publication bias, was −0.98 (95% CI = −1.09, −0.87).

Table 5.  Egger Regression Intercept Test for Potential Publication Bias for Immediate Postconcussion Assessment and Cognitive Testing^a and Postconcussion Symptom Scale Scores Measured Within 1 Week Postconcussion

View Fullscreen

For the samples with outcomes measured between 1 and 3 weeks after concussion, funnel-plot inspection showed potential asymmetry for ImPACT reaction time. The Egger regression intercept test for potential publication bias was not different for reaction time (β = −1.35, t₂₄= 0.84, P = .21).

Effect Sizes

Within 1 week postinjury, concussion had a small to moderate effect on construct-specific measures of speed and memory as measured by ImPACT. During the same assessment period, the ES of self-reported symptoms was larger than for cognitive declines. After 1 week postinjury, the PCSS score ES was comparable with that of cognitive declines observed for the 4 cognitive composite scores. Inspection of construct-specific cognitive declines revealed that whereas verbal and visual memory scores appeared to have larger ESs within 1 week of concussion than did speed scores (processing speed and reaction time), only processing speed was different from verbal and visual memory (ie, CIs did not overlap). These findings were comparable with those reported by Kontos et al,⁸ who demonstrated a slightly greater ES for visual memory (Hedges g = −0.25) than for processing speed (Hedges g = −0.18). After 1 week, the ESs for speed and memory were comparable.

When assessments were administered within 1 week postinjury, the differences in the PCSS score ES from the processing speed and reaction time ESs were significant (ie, CIs did not overlap), even after adjusting for possible publication bias for PCSS score and reaction time. These findings are similar to those reported by Kontos et al,⁸ who observed a greater ES for self-reported symptoms within 1 week postinjury (Hedges g = −0.23) than the ES for reaction time (Hedges g = −0.11) and processing speed (Hedges g = −0.18). A minimal overlap existed between the upper limit of the 95% CI for the PCSS score (−0.71) and the lower limits for verbal memory (−0.77) and visual memory (−0.74).

The cognitive declines reported within 1 week postconcussion were likely influenced by the inclusion of participants with symptoms. This assumption is further supported by the results of the meta-regression, in which a large proportion of between-studies variance for verbal and visual memory was explained by postinjury PCSS scores. These findings suggest that athletes who are acutely symptomatic demonstrate greater memory declines than those who are less symptomatic or symptom free as they recover from injury. The work of Nelson et al⁶ further supports these findings: when symptomatic and asymptomatic patients were considered together, the sensitivity of ImPACT scores to the acute effects of concussion within the first 24 hours ranged from 24.4% to 39.5%. However, the sensitivity of ImPACT diminished to 5.2% when only symptom-free athletes were considered during the same period.⁶

The trend toward a greater ES for PCSS score changes (Hedges g = −0.81) within 1 week of concussion than for cognitive declines (Hedges g = −0.43 to −0.67) was consistent with previous meta-analyses.^7,9 When initial postconcussion testing was conducted within 7 days postconcussion, Broglio and Puetz⁷ reported a greater ES for self-reported symptoms (Hedges g = −3.31) followed by cognitive declines (Hedges g = −0.70). Similarly, Dougan et al⁹ reported larger ESs for self-reported symptoms (Hedges g = −0.66) than for cognitive declines (Hedges g = −0.40). The range of within-1-week cognitive declines in ImPACT scores that we reported (−0.43 to −0.67) was larger than the overall ES that Kontos et al⁸ reported for ImPACT score (Hedges g = −0.19; P < .05). These differences may be attributed to differences in the reviewed studies or the ES computation method. In addition, we calculated the ES only when postinjury scores were compared with baseline scores; Kontos et al⁸ reported ESs in mixed studies of control group and baseline comparisons.

After 1 week postinjury, the ES for the PCSS score was comparable with the ESs of ImPACT cognitive scores, as indicated by the overlap between the CIs for all of the ESs. The range of ESs for observed cognitive declines in ImPACT scores (−0.25 to −0.37) was slightly higher than the aggregated ES (Hedges g = −0.19) for CNTs that Broglio and Puetz⁷ reported within the first 14 days postconcussion.

Meta-Regression and Unexplained Between-Studies Variance

For the meta-regression of the analysis within 1 week of concussion, the emergence of time since injury as a univariate moderator to explain the between-studies variance for all the ImPACT composite scores is consistent with previous works.^6,7,9 For instance, Dougan et al⁹ reported that the cognitive score ES decreased from −0.76 within 24 hours postconcussion to −0.44 after the first day postconcussion. However, within 1 week postconcussion, time did not moderate the PCSS score. These findings may be explained by the biopsychosocial nature of symptoms in which the perceived severity of symptoms is affected by many factors and is not perfectly related to the severity of injury or time since injury.^53,54

For the meta-regression of the analysis for 1 to 3 weeks after concussion, the emergence of time since concussion as a factor to explain between-studies variance in PCSS scores but not ImPACT composite scores was opposite to the trend observed in the within-1-week analysis. This difference may be explained by the fact that symptoms are usually more delineated after the acute period of concussion, leading to different symptom trajectories that may not be reflected in the total PCSS score.⁵⁵ In addition, these findings could be explained by the small number of studies included in the analysis, which may have underpowered the analysis of cognitive scores. For example, the post hoc analysis revealed that the power for the time regression coefficient was 24% and 6% for verbal memory and reaction time, respectively. As such, the association between a greater preinjury PCSS score and a smaller ES for reaction time beyond the first week was unexpected and may be a spurious finding (ie, false positive). An alternative explanation could be that greater preinjury symptoms are associated with worse baseline reaction times, which may have led to smaller differences between baseline and postinjury scores (ie, smaller ES). Given that no researchers have examined baseline associations between symptoms and ImPACT scores, this explanation remains speculative.

The meta-regression results did not support sex or concussion history as moderating factors for concussion effects on speed and memory at the 2 time points. The lack of association between ES and multiple previous concussions was expected because ImPACT was not designed to detect subtle cumulative declines in cognitive function attributed to multiple concussions. These findings may also be attributed to the small number of studies and the lack of details regarding concussion history, which warrant a cautious interpretation of these results. The lack of sex effects in this review was comparable with that of previous investigators^39–41 who did not find differences in ESs as a function of sex for verbal memory, processing speed, and reaction time postconcussion. However, the findings of visual memory differed from those reported by Covassin et al,^39–41 who observed that female participants experienced a greater decline in visual memory than did male participants. Moreover, the lack of sex effects differs from the report of Dougan et al,⁹ who noted sex as a moderator for neurocognitive outcomes in the first 10 days postconcussion. The difference could be explained by the more stringent criteria in our meta-analysis; we compared participants' postinjury and baseline performances on 1 specific CNT (ie, ImPACT), whereas Dougan et al⁹ examined an aggregated neurocognitive ES across multiple paper-and-pencil and computerized tests. Our stringent criteria may have limited the power needed to document possible sex effects because of the small number of studies in our review (n = 17) compared with the 78 studies in Dougan et al.⁹

Limitations

Our review had some limitations in search strategy and the meta-analytic process. We included only participants whose postconcussion ImPACT scores were compared with their own baseline scores. Therefore, the cognitive effects of concussion could be different when compared with a control group rather than the individual's own baseline. We examined only ImPACT and PCSS scores, and these findings cannot be generalized to other cognitive tests not included in this review. The authors of 3 studies^40–42 included in this review did not report an operational definition of concussion, and the definition of concussion in the remaining 16 studies varied, which may have influenced the observed ESs that we reported.

The results for concussion history and sex as possible moderators for the effects of concussion should be interpreted with caution. Concussion history was self reported and, therefore, may have been subject to recall bias. In addition, the findings about a sex effect may be limited due to the predominance of male participants (70%) in the included studies (Table 1). Whereas the included studies addressed postinjury performance relative to the individual's own baseline, the authors of most studies did not examine whether the observed cognitive decline exceeded the expected change attributed to measurement error. Without this analysis, investigators would not be able to confirm if postinjury cognitive decline could be ascribed to concussion or was a false positive attributed to the expected measurement error.¹⁸ Although multiple forms of ImPACT were developed to overcome a possible practice effect, emerging evidence⁵⁶ has indicated that some forms are not equivalent. Nonequivalence among some forms may have contributed to between-studies variance, but the effect of the forms could not be quantified because most authors did not specify the ImPACT form used.

In most previous meta-analyses, researchers have employed different methods to calculate the ES for within-participant comparisons and have not accounted for the relationship between the baseline and postinjury scores. We estimated correlations for preinjury to postinjury outcome scores for each study as recommended by Morris and DeShon,²⁵ who suggested that sampling variance may be biased by the assumption that the time effect is the same across all studies. This adjustment for repeated measures is assumed to allow for previously published values that may have been obtained from independent groups.

The precision of the ES estimates across the 5 ImPACT outcome measures should be interpreted with caution. Whereas the ESs in this meta-analysis were estimated independently for each outcome, the stochastic dependence between separate outcomes taken from the same sample (ie, convergence) may bias the statistical inference.⁵⁷ Van den Noortgate et al⁵⁸ reported that this inter-item dependence may result in underestimated standard errors, narrow CIs, and an inflated type I error rate. Statistical consideration for this potential source of bias in ES estimation requires consideration for the inter-item covariance across the studies. However, the authors of the reviewed studies did not report these correlations, and the ImPACT user's manual⁵⁹ described these relationships only for the first version of the test, which was excluded from this review. Therefore, this meta-analysis did not account for possible dependence among the observed ESs.