Authors' Response

Dear Editor:

We thank the letter authors for taking the time to review and analyze our work. We respect the points raised by the authors and agree that the method they discussed that could have been used for our study is an alternative to our established approach. However, as with many research questions, multiple valid methods are often appropriate for collecting, processing, and interpreting data. The letter authors indicated that they were primarily concerned that our methods may have increased the potential for type 1 error (ie, suggesting that areas of the brain were active when they were not). We provide an evidence-based justification for using whole-brain analysis (ie, exploratory analysis) and the subsequent secondary analyses^1–3 to answer the research questions in our study.

MULTIPLE COMPARISONS AND STATISTICAL INFERENCES

First, suggesting that on the order of 130 000 statistical tests be performed indicates a lack of understanding of the actual dimensionality of the data. In a random-effects analysis of activation from a group or groups of participants, the number of unique tests is limited to (at most) the number of individuals in the study, essentially a number that is many orders of magnitude smaller. This is fundamental linear algebra based on the rank of the data matrix. Individual voxels do not represent uncorrelated variables but are aggregated due to strong spatial correlations among them. This underlying aggregation is visible in the group activation maps, as spatially adjacent voxels tend to have a stronger correlation and may materialize as clusters or blobs. Although correlations are strongest between adjacent voxels within a cluster of activation, correlations among physically remote regions of the brain exist as well, as evidenced by studies of functionally connected brain networks.⁴ Thus, it is not clear what may constitute unique tests and multiple comparisons in an image dataset. Bonferroni and Sidak corrections are valid in the case of independent uncorrelated variables but tend to be overly conservative for correlated variables.⁵

Moreover, in Tables 2 and 3, our legends clearly stated that the P values were uncorrected, which provides transparency for readers.⁶ The analyses in our study were considered exploratory due to the novel application of these techniques in the field of sports medicine and the small sample size. It is also critical to reduce the potential for type II error as strict statistical corrections that increase this risk are often debated in the literature.⁷ When type II errors occur, viable treatment options and information that could help patients become unavailable and are potentially permanently eliminated as options for enhancing care.⁷ Additionally, as described in 1 of the letter authors' references, if exploratory data analyses are performed, the results must be presented with appropriate disclaimers, which we did.⁸ We were transparent about the exploratory nature of this study and recognized the barriers of obtaining a sample size needed for such rigorous analyses by using a novel and typically expensive measurement technique, such as functional magnetic resonance imaging (fMRI). Therefore, we opted to use uncorrected P values in Tables 2 and 3.⁶

DID CONSERVATIVE SCIENCE KILL THE CAT? OR RATHER, THE SALMON? WHAT ABOUT TYPE II ERRORS?

The letter authors are either misunderstanding or perhaps misrepresenting the referenced work on circularity. Kriegeskorte et al⁸ described circularity as using the same data to delineate regions of interest (ROIs) that will eventually be analyzed as part of the study (ie, functionally defined ROIs), in contrast to using a standardized atlas of identified brain regions. Unfortunately, a major limitation to using functionally defined ROIs is that this method lends itself to a self-fulfilling prophecy, as follows: determining a posteriori the probability of detection for a cluster of voxels that have already been identified by the investigator using the same data that demonstrated activation to the task. Now, that is “circular.” In contrast, our analysis was based on a priori-defined ROIs from an established atlas (automated anatomical labeling [AAL]).⁹ Therefore, the ROI data for the AAL atlas parcellation were extracted without any correction or thresholding. As a result, the ROI means simply represented the average of the activation measures by fractional signal change across all voxels within the previously established ROI.

We would argue that second-level analyses of activation based on a priori-defined anatomical ROIs, such as those of the AAL atlas, are in fact very informative. In using ROIs involving all voxels without thresholding, any effects that appear as statistically significant in a random-effects analysis are less likely to lead to a type I error, as this analysis expects additional random variation in the variables. An atlas-based approach also makes the results more readily interpretable, as the AAL atlas is included in most fMRI analysis and visualization packages. These ROIs are anatomically defined and do not adhere to functional region boundaries unique to the data being studied, thereby minimizing the risk of bias in the subsequent analysis.

Furthermore, a random-effects analysis of measures of activation (averaged across all voxels within an ROI) from individual participants was performed in a fully transparent manner offline using commonly available statistical software (SPSS; SPSS Inc). This method is an appropriate way to handle the data and allows us to begin to integrate traditional sports medicine statistical analyses that are more interpretable and digestible to our target audience (clinicians treating patients with anterior cruciate ligament reconstruction [ACLR]). As we analyzed between-groups and within-group differences in average fractional signal change by using traditional sports medicine statistical analyses (ie, independent and paired t tests), the “dead salmon” argument raised in the letter is not applicable to our work. Moreover, in the dead salmon argument, the authors performed a fixed-effects analysis of the time series response data from a single fMRI scan in a single (ie, the dead salmon) rather than a fully transparent random-effects analysis of activation in a group or groups of (living) participants, as in our study.⁶ In the 3-voxel cluster in the single dead salmon, type I error was, without a doubt, present. However, as eloquently argued by Lieberman and Cunningham,¹⁰ if 16 dead salmon were scanned and analyzed, the same false alarm would not be present in the same location; therefore, a group-level analysis will not show this effect as data aggregation allows for self-correcting of false alarms (ie, type I error).¹⁰ Conservative thresholding techniques, such as those proposed by the letter authors, can consequently increase type II error rates and limit our ability to advance patient care.¹⁰

Lastly, we would like to state that the regions identified were not “cherry picked.” We selected regions that were previously identified by researchers as meaningful emotional regulation areas in patients with musculoskeletal injuries³ and areas specifically different for patients after ACLR.¹¹ In fact, the development of a working hypothesis based on the most rigorous, available evidence and then testing the hypothesis are fundamental to the scientific method as first described by Aristotle. We note with strong emphasis that the regions we selected were also identified in a recent thesis¹² and associated published conference abstract¹³ on which 1 of the letter writers was the senior author, further strengthening our rationale for selecting the specific brain regions included in our study. In this work, which identified neural activity associated with kinesiophobia after ACLR, the authors used the analysis proposed in their current letter and reported similar results to ours, even though they used a different task-based paradigm.^12,13 Specifically, Kim et al¹² observed greater activation in the left thalamus, precuneus, primary somatosensory cortex, primary motor cortex, lingual gyrus, superior parietal lobe, corticospinal tract, left cerebellum, and corpus callosum. These findings align very closely with our results presented in Figure 2 and Table 2⁶ and further suggest the robust nature of our conclusion that injury-related fear may have led to altered neural patterns associated with emotional regulation in patients post-ACLR. Although the potential for type I error cannot be eliminated, we used rigorous methods to greatly minimize the potential and have since identified findings from researchers in other laboratories that corroborate ours.

TREATMENT OF TASK CONDITIONS

We used a picture imagination task paradigm similar to that in a previous fMRI study³ of emotions in patients with chronic musculoskeletal pain that demonstrated comparable results. We wanted to (1) identify whether between-groups differences existed regardless of picture category and if they did exist (2) begin to identify whether we could quantify if it was because of sport-specific or activities of daily living images. We could have also completed this analysis in our healthy control participants, but it was unnecessary to answer our specific questions for patients post-ACLR. Our interest was not in whether this was present in healthy patients; therefore, we only conducted this in our population of interest. We invite colleagues in the field, including the letter authors, to replicate our study by using the proposed picture imagination task paradigm to examine whether differences are present in healthy individuals with respect to sport-specific images or activities of daily living images.

CONCLUSIONS

The purpose of our investigation was to provide further contextual evidence that injury-related fear should not be overlooked during rehabilitation. However, the letter authors attempted to nullify our findings when our methods were sound and our results suggested that an area of care is lacking in patients post-ACLR (ie, psychological evaluation and rehabilitation). Activation patterns similar to what we reported (ie, increased activation in the thalamus, cerebellum, and occipital regions and inability to suppress the default mode network) have been observed in patients with chronic musculoskeletal pain,³ with medial patellofemoral ligament deficiency,¹⁴ and after ACLR⁷ by using the analyses described by the letter authors. Subsequently, our findings support a growing body of literature in which researchers emphasize that injury-related fear may lead to objectively measured brain changes after musculoskeletal injury and warrant targeted interventions. We hope that the use of neuroimaging in sports medicine research continues to grow with both new and established investigators conducting novel hypothesis-generating studies aimed at solving significant clinical problems.