False-Positive Rates, Risk Factors, and Interpretations of the Vestibular/Ocular Motor Screening in Collegiate Athletes

Vestibular dysfunction is one of the most reported symptoms after a sport-related concussion (SRC), with up to 84% of high school athletes reporting dizziness after injury.^1,2 Vestibular dysfunction may also manifest as vertigo, nausea, postural instability, and visual problems. Similarly, 81% (n = 200 of 247) of adolescents between 7 and 18 years of age who presented with at least 1 vestibular symptom after a diagnosed concussion took on median 53 days longer to return to school and 77 days to return to sport when compared with pediatric athletes without the same symptoms.³ The sizable differences in time removed from school and sport emphasize the importance of evaluating for vestibular deficits after SRC.

The Vestibular/Ocular Motor Screening (VOMS) was developed to assess vestibular and ocular motor impairments via symptom provocation.⁴ Previous clinical interpretation guidelines suggest that having any VOMS item symptom score greater than or equal to 2 (2 of 10) and a near-point convergence (NPC) distance greater than or equal to 5 cm increases the probability of correctly identifying young athletes with concussion by 50% and 38%, respectively.⁴ More recently, a cutoff criterion that consists of a post-VOMS symptom score greater than or equal to 8 was established in collegiate athletes.⁵ A singular criterion of a symptom score provides clinicians with a simple and clear means for VOMS interpretation.⁵ Independent evaluation of false-positive rates and contributing factors is warranted for this interpretation criterion in collegiate athletes.^6,7

The reliability, discriminative validity, concurrent validity, and change scores of the VOMS have been previously described; however, as the clinical interpretation guidelines of the VOMS are variable, there is no general consensus on which guidelines to use.^4,5,7–14 The existence of multiple interpretation guidelines and a lack of consensus related to which criteria to use to interpret the VOMS is problematic. Using the interpretation criteria of a symptom score greater than or equal to 2 for any VOMS subtest and an NPC distance greater than or equal to 5 cm, the VOMS has been demonstrated to have a low false-positive rate of 11% in collegiate athletes during the preinjury (baseline) concussion assessment.⁷ The false-positive rate is doubled (22%) in military personnel using these same criteria.⁶ In adolescent athletes, false positives have ranged from approximately 9% to 29.3%.^9,11,15 Potential rationale for these false-positive rates include a history of motion sickness and migraine as well as an inability to accurately describe personal symptomology.^6,7,9,16,17 Though the evidence related to biological sex on recovery from concussion remains mixed, authors of previous studies have suggested that females are more likely to report suspected concussions and symptoms, which may contribute to a false-positive VOMS.^18–20 For example, female collegiate athletes are 3 times more likely to exhibit a false-positive VOMS than males.⁷

Given that motion sickness and biological sex are associated with a false-positive VOMS in healthy collegiate athletes, additional modifiers may confound the results of the assessment. The identification of additional modifiers is warranted to assist clinician understanding of potential sources of error that may influence the clinical interpretation of the VOMS before and after concussion. Moreover, the newly established clinical cutoff criteria in collegiate athletes have yet to be vetted in terms of false-positive rates in an independent sample.⁵ Therefore, the purposes of this study were to (1) determine the false-positive rate at the baseline assessment, (2) identify additional risk factors that may contribute to a false-positive VOMS, and (3) compare the false-positive rate with other published clinical interpretation guidelines.

METHODS

RESULTS

Demographics

Of the original 511 National Collegiate Athletic Association Division I collegiate athletes, 42 were excluded due to missing data on one or more of the primary outcome measures, whereas 7 declined to participate. Therefore, a total of 462 (187 female [40%], 275 male [60%]) collegiate athletes were included in our analyses. Groups were similar in age (true-negative = 18.8 ± 1.4 years vs false-positive = 18.5 ± 1.1 years, P > 0.05) and other demographic variables (Table 1). Medians and interquartile ranges for each continuous risk factor included in our model across groups are presented in Table 2.

Table 1. Medical History Demographics per Group

View Fullscreen

Table 2. Medians and Interquartile Ranges (IQRs) for Continuous Risk Factors per Group

View Fullscreen

False-Positive Rates

Approximately 9.1% of participants (12 males, 30 females) demonstrated a false-positive VOMS assessment. Among participants with false positives, there was a significantly greater proportion of females (n = 462, χ² = 18.37, P < .001, 4% males vs 16% females). Table 3 summarizes the false-positive rates in the current study as well as previously reported false-positive rates in relationship to different cutoff criteria and samples. Table 4 demonstrates how false-positive rates can vary because of different cutoff interpretations, using the data from this study.

Table 3. Previously Established Vestibular/Ocular Motor Screening False-Positive Rates

View Fullscreen

Table 4. False-Positive Rates Associated With Different Cutoff Interpretations in Current Study

View Fullscreen

Multivariable LR

The overall binary LR was statistically significant (P < .001) and accounted for about 36% of the total variance (R² = 0.355). Sex (OR = 2.79; 95% CI = 1.17, 6.65; P =. 020) and history of treatment for headache (OR = 4.99; 95% CI = 1.21, 20.59; P = .026) were the only significant predictors of a false-positive VOMS assessment. Results of the LR model are presented in Table 5.

Table 5. Results of Binary Logistic Regression

View Fullscreen

DISCUSSION

The primary aim of our study was to cross-validate population-specific VOMS clinical interpretation guidelines. A secondary aim of this study was to determine contributing factors to a false-positive VOMS based on these and other population-specific interpretation criteria (Tables 3 and 4).^{6,7,11,13,16,37} The overall false-positive rate using newly established cutoff criteria in collegiate athletes was 9.1%, which is consistent with prior research.⁷ We observed that being a female and receiving prior treatment for headache were associated with an increased risk for a false-positive VOMS. Lastly, we determined that false-positive rates are largely dependent on interpretation guidelines. Our findings should be considered when administering the VOMS, especially in the absence of a baseline assessment.

Reported false-positive rates of the VOMS range from 9% to 29.3% in adolescents to 21.9% in US military personnel and 11% in collegiate athletes based on criteria established in young athletes.^6,7,9,11 For this study, we applied an interpretation criterion (≥8 total symptoms) specific to collegiate athletes.⁵ Using this criterion, our sample of healthy collegiate athletes had a low false-positive rate of 9.1%, which is consistent with previous research.^6,7,9,11 Given the low false-positive rate observed in this study and the previously published high area under the curve (0.91), clinicians that work with collegiate athletes should consider using the total symptom score (≥8) to interpret the VOMS.⁵ This is the first study to cross-validate the cutoff score reported by Kontos et al in concussed collegiate athletes and is in partial alignment with the recently published Sport Concussion Office Assessment Tool-6, as it does not include the NPC.^5,38 In sum, using a total symptom score (≥8) is a relatively easy way to interpret the VOMS with an acceptable false-positive rate and in conjunction with the high discriminative ability found previously in collegiate athletes.⁵

Biological sex and a history of motion sickness are known risk factors for false-positive VOMS performance in collegiate athletes. Kontos and colleagues reported that 72% of athletes who exceeded clinical cutoffs reported a history of motion sickness and that female athletes were 3 times more likely to have one or more VOMS items above the clinical cutoff, leading to a false-positive outcome.⁷ Similarly, Henry et al observed that males reported fewer vestibular-ocular symptoms when using the VOMS within 2 weeks of a diagnosed concussion.³⁹ Given the consistency of biological sex differences with this study and related research associated with vestibular-ocular function after concussion, we believe this warrants further discussion.

As mentioned in our introduction, the influence of female sex on recovery from concussion remains equivocal.¹⁸ That said, some evidence suggests that high school female athletes are more likely to develop persisting symptoms (ie, lasting greater than 28 days) after a concussion.^40–42 In this study, biological sex was identified as a key factor associated with a false-positive assessment. This finding may be due to females being more likely to report symptoms.⁴³ Brown et al concluded that healthy female athletes were more likely to report vision or hearing problems, headache/migraine, and difficulty concentrating during baseline assessments.⁴³ These symptoms are consistent with those of the VOMS. That female athletes are more willing to disclose symptoms may be a result of psychology but also of physiology in the absence of a concussion. For example, the menstrual cycle may be associated with vestibular-ocular symptom reporting in the absence of a concussion. Symptoms such as headache and difficulty concentrating have been associated with premenstrual syndrome and are similar to the VOMS symptoms headache and fogginess.⁴⁴ Nausea, a symptom recorded as part of the VOMS, and motion sickness are predictors of a false-positive VOMS. Each symptom has been demonstrated to be significantly higher in collegiate females who do not use oral contraceptives during symptom provocation tasks similar to the VOMS.⁴⁵ Accounting for the menstrual cycle at the baseline and postinjury assessments is important and may assist clinicians in determining if symptoms provoked during the administration of VOMS are due to vestibular-ocular dysfunction or another cause. Future researchers should consider the development of sex-specific cutoffs to ensure that clinicians are providing individualized care to their patients in accordance with the biopsychosocial model of health care.

Interestingly, a history of treatment for headaches was also a significant predictor in this study; however, the confidence interval was relatively large and thus this result should be interpreted with caution. Despite the inclusion of multiple predictors in our regression model, no other modifiers were identified. Surprisingly, the SOT outcome scores were not associated with false-positive VOMS assessments. The SOT was included as a predictor in our analysis as the vestibulo-ocular and vestibulospinal systems are challenged during the assessment. Though the SOT outcome scores were not associated with a false-positive VOMS, this does not imply that the VOMS and SOT are always measuring different constructs. However, authors of 2 separate studies concluded that the VOMS measured different aspects of vestibular function when compared with the Balance Error Scoring System and King-Devick assessments, which also purport to measure vestibulo-ocular and vestibulospinal function.^4,10 Future researchers should investigate the relationship between the SOT and VOMS outcome scores.

Lastly, we compared different VOMS cutoff interpretation guidelines in terms of false-positive rates. To date, the most frequently used clinical cutoff to evaluate VOMS performance has been a symptom score greater than or equal to 2 for any domain or an NPC of greater than or equal to 5 cm.⁴ However, discrepancies exist in terms of the wording of these clinical interpretation guidelines. The underlying issues associated with variable clinical interpretation criteria are based on the use of and, or, and and/or. For example, when discussing the clinical cutoffs in their abstract, Mucha et al stated: “An NPC distance ≥ 2 5 cm and any VOMS item symptoms score ≥ 2 resulted in an increase in the probability of correctly identifying concussed patients of 38% and 50%, respectively.”⁴ In their conclusion, Mucha and colleagues stated that “cutoff scores of ≥ 2 total symptoms after any VOMS item or an NPC distance of ≥ 5 cm resulted in high rates of identifying concussions.”⁴ Although those sentences technically mean the same thing, it may lead to confusion in the clinical interpretation of the VOMS. Similarly, articles have reported using the symptom score and/or as well as or NPC (Table 3).^6,7,9,11,16 For example, authors of 1 study explicitly mentioned using greater than or equal to 2 symptoms and/or a NPC of greater than or equal to 5 cm; however, in the same article the authors reported that “clinical cutoff scores are defined as ≥2 symptoms on VOMS subscales or ≥5 cm on NPC distance.”⁹ These criteria can be interpreted in 2 distinct ways:

1. An individual would need to be above both cutoffs to be considered positive.

2. An individual would need to be above only one cutoff to be considered positive.

These alternate interpretations based on and/or versus just or may change interpretation and decision-making, though the VOMS should not be used in isolation for clinical decision-making after a concussion.

A separate study reported a false-positive rate of 11% based on symptom scores less than 2 on all VOMS items; however, the authors seemed to have included the NPC distance in their original operational definition of what constituted a false-positive.⁷ A discrepancy exists between the results found in Table 3 within that manuscript, as the table shows 11% based on NPC distance only, not symptoms.⁷ Another discrepancy in the literature related to the VOMS includes interpreting the cutoffs as a function of a symptom provocation change score from baseline, despite the Mucha et al article’s not accounting for a change from the baseline assessment.^4,46,47 Additionally, in terms of the NPC distance cutoff, researchers have reported using both greater than and greater than or equal to 5 cm, sometimes even within the same study.^6,16Table 5 provides examples of how these different verbiages influence the false-positive rates when applied to the data of this study. Overall, these observations reflect small, but important, changes and inconsistencies in verbiage that can make it difficult to interpret the VOMS and to compare findings across multiple studies. Our findings support a simple and cross-validated way to interpret the VOMS.⁵

This study is not without limitations. As with any study involving patient-reported outcomes, recall bias may have influenced our demographic results. It is unclear whether the order of tests, which was relatively random based on clinician availability, may have influenced our results. For example, it has been shown that strenuous exercise can influence VOMS results.⁴⁸ However, our participants were tested in small groups of no more than 4 at a time (tested individually during the VOMS assessment) by 1 of 3 clinicians who were all trained in the administration of the VOMS by the same individual. The SOT was administered on 1 of 2 balance units due to the discontinuation of one (Natus) and an upgraded unit (Bertec) being installed during the study. It is unclear whether the use of 2 different units administering the SOT had any significant effect on the SOT outcome scores, as the Bertec Computerized Dynamic Posturography unit’s measurement properties have yet to be established in collegiate athletes. That said, the SOT guidelines and algorithms are the same between the 2 units. Additionally, we did not query athletes about a history of motion sickness, a known risk factor that may also have contributed to misclassification.^6,7 As this study is part of an ongoing prospective study, motion sickness was not queried because the ongoing study was not specific to false-positive VOMS assessments a priori; however, the addition of an item asking about a history of motion sickness is warranted if the screen is to be used. Said again, as we did not evaluate the role of motion sickness as a risk for a false-positive VOMS, we cannot support or refute the findings of related literature. Regarding our LR model, there were relatively few individuals in the false-positive group given the number of predictors entered. Importantly, balancing our data is not necessarily the only valid approach in this instance. If one of the classes (ie, true negatives) is actually much more common in the population (which it is according to a variety of other VOMS studies examining false-positive rates) and not solely in our sample, a naïve model (ie, classifying everything as belonging to the most common category [ie, true-positive]) can actually be a good estimate. In other words, although there are few false-positive cases, we feel that the model provides a good idea of what “normal” is, given the large number of true negatives. Additionally, history of headache as a predictor has a wide confidence interval and so that finding should be interpreted with caution. Given that the only other predictor was sex, and that sex differences have been found in previous work, we feel that, although the model might not be perfect, it gives a good picture of what might classify someone as false-positive. Lastly, we did not assess interrater or intrarater reliability of VOMS assessors; however, all assessors were certified athletic trainers who were instructed by the same individual in the administration of the VOMS.

CONCLUSIONS

When using VOMS interpretation guidelines specific to collegiate athletes, the false-positive rate was similar to that found in previous research. Female athletes and athletes with a history of treatment for headaches had a higher false-positive rate than those who did not. Biological sex and prior treatment for headaches should be considered when interpreting the VOMS before and after a concussion. Understanding what an athlete is “trying to get back to” after a concussion when using the VOMS coupled with the ability to detect potential vestibular dysfunction in the absence of concussion emphasizes the importance of the VOMS baseline assessment if it is to be used clinically. Our findings support using population-specific interpretation criteria for the VOMS, as false-positive rates vary when alternate interpretation guidelines are used. Lastly, clinicians and researchers should consider their verbiage when incorporating the VOMS into concussion protocols for the purposes of simplified and consistent interpretation. The use or misuse of and, or, or and/or may change the interpretation of the VOMS and influence clinical decision-making after a concussion.