Assessing the Validity of the Mental Health–Related Survey in Collegiate Student-Athletes
Context
Mental health screenings are recommended during preparticipation physical examinations. The Mental Health–Related Survey (MHRS), a 9-item questionnaire adapted from the 18-item Mental Health Screening Form-III, is suggested in 3 consensus and/or position statements. However, there is no evidence on the effectiveness of the MHRS.
Objective
To assess the validity of the MHRS for mental health screening in collegiate student-athletes.
Design
Cross-sectional study.
Setting
University athletic program.
Patients or Other Participants
Five hundred fifteen National Collegiate Athletic Association Division II student-athletes (age = 20 ± 1 years).
Main Outcome Measure(s)
Participants completed the MHRS, the Patient Health Questionnaire for depression (PHQ-9), and the Generalized Anxiety Disorder 7-Item Scale (GAD-7) for anxiety. A stratified sample underwent a neuropsychiatric interview (Mini-International Neuropsychiatric Interview). Descriptive statistics and Pearson correlations were performed. An area under the curve analysis compared the MHRS with the Mini-International Neuropsychiatric Interview. Validity was determined using sensitivity, specificity, Youden index, predictive values, and accuracy.
Results
Three hundred twenty-two student-athletes (62.5%) indicated yes to 1 or more items on the MHRS, suggesting they would require a mental health referral. Women indicated more yes answers than men (P < .001). Average scores were 2.21 ± 3.06 on the PHQ-9 and 2.66 ± 3.87 on the GAD-7. Using a cut score of 6, 68 individuals (13.2%) reported clinically relevant depression, and 76 (14.8%) reported anxiety. The PHQ-9 and GAD-7 scores strongly correlated with MHRS scores (PHQ-9: r = 0.713, P < .001; GAD-7: r = 0.745, P < .001). The MHRS had a specificity of 24.6%, a sensitivity of 93.9%, and overall accuracy of 40.14%. The area under the curve score was 0.762. We identified a new cutoff score for the MHRS of ≥4; however, the sensitivity of 63.6% and specificity of 76.3% raise concerns regarding how well this tool can rule out and in clinically significant symptoms of mental health conditions.
Conclusions
Most student-athletes indicated yes to at least 1 item on the MHRS, warranting a mental health referral. The MHRS showed high sensitivity but low specificity, indicating low clinical utility as a screening tool.
Key Points
Although a majority of collegiate student-athletes (62.5%) indicated at least 1 symptom on the Mental Health–Related Survey (MHRS) that would have warranted a referral, only 13.2% reported clinically relevant depression, and 14.8% reported clinically relevant anxiety.
The MHRS has a specificity of 24.6% and a sensitivity of 93.9%, with an overall accuracy of 40.1% compared with a neuropsychiatric interview.
Based on our data, the MHRS, a 9-item questionnaire recommended in 3 consensus or position statements, has low clinical utility and is not recommended for preventive mental health screening.
The preparticipation physical evaluation is critical in identifying collegiate student-athletes’ preparedness for safe participation in sports.1 Mental health, as an influential factor of this preparedness, should be carefully assessed in this population during the preparticipation physical evaluation, given the existing prevalence rates of depressive and anxious symptomatology among collegiate student-athletes. The National Collegiate Athletic Association (NCAA), in collaboration with the American College Health Association, identified that 20% of female, 13% of male, and 40% of nonbinary collegiate student-athletes reported debilitating depression, and 41% of female, 22% of male, and 50% of nonbinary collegiate student-athletes reported overwhelming anxiety.2 Failure to identify these mental health concerns before, during, or after their respective sports seasons may lead to a deterioration in their overall health and the occurrence of mental health emergencies such as suicide attempts.3 For this reason, the second edition of the Inter-Association Consensus Document on Mental Health Best Practices recommends that the interdisciplinary health care team for collegiate student-athletes, which should include athletic trainers (ATs), use validated mental health questionnaires to screen their student-athletes at least once annually, with validated being defined as “[the existence of] reputable scientific evidence that the tool measures what it says it is supposed to measure.”4
The National Athletic Trainers’ Association (NATA) position statement on preparticipation physical evaluations and disqualifying conditions echoes the sentiment that mental health status questions should be considered during the health history portion of the preparticipation physical evaluation.1 The position statement recommends using the Mental Health–Related Survey (MHRS). The MHRS is also a recommended screening tool in the Inter-Association Consensus Statement on Psychological Considerations for student-athletes within the collegiate5 and secondary school settings.6 The MHRS is a 9-item, binary (yes/no) tool adapted from the 18-item Mental Health Screening Form-III (MHSF-III). Interestingly, the MHSF-III was initially designed as a nonspecific mental health questionnaire for chemically dependent persons seeking admission to substance abuse programs.1,7 To date, there is no readily available evidence for ATs on the processes through which the MHRS was adapted from the MHSF-III. There is also no existing information on the validation or reliability of the MHRS in identifying collegiate student-athletes with mental health concerns since its inclusion in the aforementioned position and consensus statements from 2013, 2014, and 2015.1,5,6 A 2016 publication highlighted the concern surrounding the validity and appropriateness of the MHRS in the athlete population and called for further examination of the patient-reported outcome measure.8
The health care team, including the AT, should feel confident in the questionnaires that they integrate as part of their mental health screening processes to appropriately and efficiently identify, follow up with, and refer any student-athletes who are struggling with their mental health. Specifically, the MHRS needs to be evaluated for how well it can identify collegiate student-athletes with mental health symptoms in comparison with other validated questionnaires of depression and anxiety symptoms.8 The purpose of this study was to examine the validity of the MHRS as a tool in the mental health screening of collegiate student-athletes. It was hypothesized that MHRS would perform with a high sensitivity, given that its items are broad enough to collectively encapsulate symptoms of various mental health conditions, but with a low specificity, due to the phrasing for many of its items being attributed to routine experiences associated with participation in college and/or sports.
METHODS
Study Design
In this cross-sectional study, we assessed the validity of the MHRS in collegiate student-athletes. The study involved administering the MHRS alongside the Patient Health Questionnaire for depression (PHQ-9) and the Generalized Anxiety Disorder 7-Item Scale (GAD-7) with a follow-up assessment guided by the Mini-International Neuropsychiatric Interview (MINI) for assessment of mental health disorders. Independent variables included sex, sport, and meeting the criteria for mental health conditions using the MINI; the dependent variable was the total MHRS score.
Participants
We invited all collegiate student-athletes from a single NCAA Division II university in the northeastern United States to participate during the in-person fall preparticipation physical evaluation. Collegiate student-athletes under 18 years of age were excluded. Institutional ethical review board approval was obtained before the study.
Instrumentation
The MHRS
The MHRS is a 9-item, binary questionnaire with yes/no response options. The MHRS was adapted from an original tool titled the MHSF-III, which was developed in 2001 for use within the chemically dependent population (ie, patients diagnosed with alcohol or substance use disorders).7 The MHSF-III was validated in 111 patients undergoing treatment within an inpatient therapeutic community for individuals with substance use disorders.9 However, the 9 items in the MHRS, which are displayed in the position and consensus statements,1,5,6 do not match any of the original MHSF-III questions. Although these position and consensus statements indicate the MHRS was adapted with permission from Alcohol Treatment Quarterly, the publication source of the original MHSF-III survey, it is unclear how the adaptation or changes to the MHSF-III occurred.
For clinical use of the MHRS within the student-athlete population, the consensus statements on psychological concerns in student-athletes recommend that “any affirmative answers in the mental health section of the preparticipation physical examination should be brought to the attention of the team physician, so that [they] may discuss them with the student-athlete and ascertain if any follow-up evaluation, care, or medication is required,”5 with similar directives in the position statement on preparticipation physical evaluations stating that “any yes answers should trigger a private discussion between the physician and athlete. The physician can then determine if the athlete needs to be referred for evaluation by a mental health care professional.”1 This suggests the MHRS has a cut score of 1 based on the referral process.
The PHQ-9
The PHQ-9 is a 9-item self-report tool validated as a reliable screening measure for major depressive disorder in the general10 and athletic population.11 The tool asks the individual to express symptom frequency from 4 options ranging from none (0) to nearly every day (4). A total score is calculated to classify depression symptom severity in the general population as follows: 0 (none), 1–4 (minimal), 5–9 (mild), 10–14 (moderate), 15–19 (moderately severe), and 20–27 (severe).10 A total score of ≥10 demonstrates 88% sensitivity and specificity compared with a structured neuropsychiatric interview for diagnosing major depression.10,12 In the present study, any participant scoring ≥6 was referred to medical personnel for clinically relevant depression based on prior validation in the collegiate student-athlete population.11 Student-athletes scoring in this range were privately consulted by the AT and offered a mental health referral. Any student-athlete indicating suicidal ideation on the final item of the PHQ-9, regardless of the total score, was automatically referred.
The GAD-7
The GAD-7 is a brief self-report tool validated as a reliable measure of anxiety symptoms in the general population.13 The tool comprises 7 items that are scored across a 3-point scale, resulting in a total score range of 0 to 21. Total scores classify anxiety severity as follows: 0 (none), 1–4 (minimal), 5–9 (mild), 10–14 (moderate), and 15–21 (severe).13 A score ≥10 demonstrates 89% sensitivity and 82% specificity in the general population compared with a structured psychiatric interview as the criterion standard for generalized anxiety disorder. In this study, a score of 6 or higher was used to indicate clinically relevant anxiety symptoms, based on validation in the collegiate student-athlete population.14
The MINI
The MINI is a brief, structured diagnostic neuropsychiatric interview designed to identify common psychiatric disorders based on the American Psychiatric Association’s Diagnostic and Statistical Manual of Mental Disorders, 5th edition (DSM-5) criteria.2,15 It was developed to provide a shorter yet accurate alternative to traditional clinical interviews for use in research and clinical settings.16 The MINI is widely recognized as a reference standard for validating mental health assessments.12,16,17 For this study, the MINI modules for major depressive disorder, generalized anxiety disorder, social anxiety disorder, and panic disorder were administered, assessing symptoms over the past 2 weeks against DSM-5 criteria. Participants meeting the criteria for any disorder were classified as positive for that condition.16 Those meeting the criteria or reporting suicidal ideation were referred to counseling services. The MINI served as the reference standard to assess whether student-athletes met the criteria for clinical depression and anxiety and to evaluate the validity of the MHRS.
Procedures
Student-athletes completed the PHQ-9 and GAD-7 as part of their annual preparticipation physical evaluation. Additionally, student-athletes completed the MHRS for the purpose of this study. A consent form was provided at the preparticipation physical evaluation that outlined the study’s purpose and ensured access to mental health screening results that were limited to ATs and researchers. Consenting participants completed the surveys electronically, including demographic questions, administered by the institution’s ATs. Names were included on the screening tools to allow ATs to identify student-athletes with clinically significant symptoms on the PHQ-9 or GAD-7. Those identified were offered referrals to on-campus counseling services. Athletic trainers were trained in mental health screening and referral protocols in alignment with the institution’s mental health management plan.
After completing the screenings, 147 student-athletes (28.5%) were selected for the MINI using a 2-phase random stratified sampling method. This approach ensured representation across the full range of PHQ-9 and GAD-7 scores, capturing participants with diverse mental health symptoms while maintaining proportional representation by sport. For a detailed description of the stratified sampling methodology, see Keenan et al.11 Graduate student clinicians from the master of counseling, social work, or school counseling programs at the participating institution were recruited as research assistants to administer the MINI. All research assistants completed coursework on clinical interviewing and participated in standardized training modules. The training was supervised by a licensed psychologist and required the research assistants to administer 2 mock MINI interview sessions before data collection. The student-athletes participating in the MINI received a $10 gift card as an incentive. The research assistants were independent of the athletics and athletic training departments and were blinded to the MHRS and other screening results.
Data Analysis
All data were downloaded and exported into SPSS (version 29; IBM Corp). We performed descriptive statistics to examine the demographics, main outcome measure scores, and the MINI. A Mann-Whitney U test was performed to compare data between genders (men and women) for the outcome variables. Two Pearson correlations were performed to explore the relationship between the MHRS and previously validated mental health screening tools (PHQ-9 and GAD-7). Next, we performed an area under the curve analysis to indicate the maximal discrimination of the MHRS between student-athletes who met and did not meet the criteria for any mental health condition using the MINI. The cutoff score for the MHRS was determined using the respective sensitivity, specificity, and Youden index. Finally, we performed a 2 × 2 contingency table for the MHRS compared with the MINI to calculate the positive and negative predictive values, false-positive and -negative rates, and overall accuracy. The significance level was set at P < .05 a priori.
RESULTS
In total, 515 NCAA Division II student-athletes (age = 20 ± 1 years; range, 18–31 years) participated in the mental health screening. The student-athletes from football represented the largest number of participants (n = 90, 17.5%), and participants were evenly distributed among school classifications. A little over 10% (n = 54) of the participants reported going through a recent (within the past 3 months) life transition or life event that they felt negatively affected their current emotional or psychological state. Full demographics are available in Table 1.
The MHRS
On the MHRS, 193 student-athletes (37.5%) indicated no on all items, and 322 student-athletes (62.5%) marked at least 1 item that warranted a referral to a licensed mental health care professional. Figure 1 displays the percentage of yes answers per participant and Table 2 provides the number of yes answers per statement. Overall, participants averaged a score of 1.60 ± 1.80 on the MHRS. A Mann-Whitney U test indicated that the MHRS total score was significantly greater (U = 21, z = −7.035, P < .001) for women (median = 2; interquartile = 0–3) compared with men (median = 0; interquartile = 0–2).
Citation: Journal of Athletic Training 60, 11; 10.4085/1062-6050-0063.25
Depression and Anxiety Screening
The participants had a mean score of 2.21 ± 3.06 (range, 0–18) on the PHQ-9 and a mean score of 2.66 ± 3.87 (range, 0–20) on the GAD-7. Overall, on these instruments, 41.4% (n = 213) and 42.5% (n = 219) of participants did not indicate symptoms warranting referral for depression and anxiety, respectively. Using the cut score of 6 on the PHQ-9 and GAD-7, we identified 68 individuals (13.2%) reporting clinically relevant depression symptoms and 76 individuals (14.8%) reporting clinically relevant anxiety symptoms on the PHQ-9 and GAD-7, respectively. Total scores from the PHQ-9 and GAD-7 had strong positive correlations with the MHRS total score (PHQ-9: r = 0.713, P < .001; GAD-7: r = 0.745, P < .001).
Neuropsychiatric Interview Findings
In total, 147 neuropsychiatric evaluations were completed, with 33 participants screening positive for at least 1 mental health condition. Seventeen participants screened positive for 1 condition, 13 were positive for 2 conditions, 2 screened positive for 3 conditions, and 1 participant screened positive for 4 mental health conditions. One hundred fourteen student-athletes did not screen positive for any mental health condition during the MINI interview. Overall, 20 individuals (13.6%) met the criteria for panic disorder, 19 (13.0%) met the criteria for GAD, 9 (6.1%) met the criteria for major depressive disorder, and 5 (3.4%) met the criteria for social anxiety disorder. Specific to major depressive disorder, 122 individuals (83.0%) reported no current or previous history of depression; however, 16 individuals had a past or recurrent depression diagnosis without current symptoms. The 9 participants who met the criteria for current major depressive disorder in this study were further classified as current only (n = 1), current and past (n = 1), current and recurrent (n = 2), or current, past, and recurrent (n = 5).
Instrument Results and Clinical Utility
The data from the MHRS were compared with the gold standard, the MINI diagnostic interview, to assess clinical utility. The Youden index for the MHRS using a cutoff score of 1 was 0.185, indicating no diagnostic value. We identified a specificity of 24.6% and sensitivity of 93.9% for the MHRS (Figure 2) with a 75.4% false-positive rate and 6.1% false-negative rate. The overall accuracy of the MHRS was 40.14%. The 2 × 2 contingency table is reported in Table 3. The positive predictive value was 26.5%, and the negative predictive value was 93.3%. The area under the curve score was 0.762, suggesting the capability for fair discrimination. Based on the Youden index, the cutoff score for mental health concerns was a total of ≥4 on the MHRS in student-athletes, corresponding to a sensitivity of 63.6% and specificity of 76.3% (Table 4).
Citation: Journal of Athletic Training 60, 11; 10.4085/1062-6050-0063.25
DISCUSSION
Our purpose was to validate the MHRS in the collegiate student-athlete population due to the emphasis placed on the tool in position and consensus statements.1,5,6 Although a majority of collegiate student-athletes (62.5%) indicated yes to at least 1 item on the MHRS that would have warranted a referral, only 13.2% reported clinically relevant depression and 14.8% reported clinically relevant anxiety. The overall accuracy of the MHRS was 40.1% when compared with a neuropsychiatric interview. We identified that the MHRS has a specificity of 24.6% and a sensitivity of 93.9% when using the cut score of ≥1, suggesting that it is a good tool to rule out those who may need a referral but has low diagnostic value to rule in and identify individual mental health concerns. Although the MHRS is recommended as a screening tool in the position and consensus documents, our data suggest that the instrument has poor clinical utility.
Screening for Mental Health Concerns
When screening for mental health conditions, such as depression and anxiety, it is ideal to use screening instruments that accurately reflect the diagnostic criteria of these conditions in order to increase the accuracy of the screening and decrease the number of false positives. Previous researchers identified that only 49% of collegiate ATs were screening for behavioral health concerns using validated tools.18 Both the PHQ-9 and GAD-7 have been found to be reliable and validated instruments for screening in an athletic population and were developed based on the DSM-5 criteria.11,14 When using these validated instruments, we identified that 13.2% of participants reported clinically relevant depression symptoms and 14.8% reported clinically relevant anxiety symptoms.
Our results demonstrated that the total scores from the PHQ-9 and GAD-7 had strong positive correlations with the MHRS total score. However, we identified a low specificity for the MHRS, suggesting that practitioners who use this instrument are more likely to receive a false-positive result. This is problematic because the medical staff will need to direct considerable resources to the follow-up process with student-athletes who are reporting well below clinically relevant symptoms. Other valid mental health screening tools, such as the PHQ-9 and GAD-7, are widely and freely available. The use of the MHRS is not recommended for mental health screening.
The Mental Health–Related Survey
On the MHRS, 37.5% of participants indicated no on all items, whereas 62.5% indicated yes on at least 1 item on the MHRS, which would warrant a referral to a licensed mental health care professional. There was a significant difference between genders, with women having a higher average score on the MHRS compared with men. This is consistent with previous research with athletes who identify as women endorsing more symptoms of mental health distress than men.19–21
The MHRS, as currently recommended (any score >1 equates to a referral to a licensed mental health care provider), has poor specificity, which may lead a health care provider to potentially place excessive demands on mental health resources. Given the high false-positive rates of the MHRS, clinicians using this survey may feel compelled to use a different cutoff or clinician utility approach to the MHRS. We identified that a cutoff score of ≥4 for the MHRS has the most clinical utility; however, the sensitivity of 63.6% suggests uncertain predictive value, and a specificity of 76.3% would suggest a likely diagnosis; a strong instrument would have a specificity and sensitivity of 90% to suggest very likely predictive value.22 Donohue et al pointed out that “consistent with this guideline, some administrators of universities have initiated procedures to augment physical safety screens with nonempirically validated assessments of mental health whereas others have incorporated mental health screening instruments that are psychometrically validated in nonathlete populations.”23(p562) Therefore, based on the data, the MHRS is not recommended for preventive screening.
The results from this study indicate that the MHRS has low diagnostic value including construct validity compared with the PHQ-9 and GAD-7, vague content validity in terms of the high false-positive rate, and weak criterion validity with an overall accuracy of 40%. These findings lead to concerns about the clinical utility of the MHRS. Clinical utility encompasses 3 main areas for health care providers: acceptability, feasibility, and appropriateness.24 The MHRS has moderate acceptability. Acceptability involves the user experience (patient and AT) and references factors such as length of the tool, time to complete, and clarity of what the question is asking the patient. The MHRS has good feasibility, as it is easy to implement and is quick to score. However, our results bring into question the appropriateness of the MHRS due to the poor metrics associated with the cutoff scores and high false-positive rates. When compared with the MINI diagnostic interview, the MHRS had a significant false-positive rate, meaning that the MHRS incorrectly indicated that student-athletes were experiencing diagnosable mental health conditions. It is important for ATs and other health care providers to use accurate and effective screening measures when assessing student-athlete mental health to provide the best care possible. It is evident that the MHRS has low appropriateness regarding the practical aspects of the patient-reported outcome in a clinical environment and may not be an adequate measure as a preparticipation mental health screener.
Limitations
Our study does have limitations worth noting. As our data collection occurred at only 1 NCAA Division II institution, results may not be accurate for other competition levels and age groups, particularly youth and secondary school student-athletes. Furthermore, we validated the MHRS against 2 commonly used mental health screeners, the PHQ-9 and GAD-7, but we did not evaluate the clinical utility of the MHRS in identifying disordered eating, insomnia, alcohol and substance use, attention-deficit/hyperactivity disorder, or other mental health disorders. Given the brevity of the MHRS and our current findings, it is unlikely that this survey would produce high specificity in discriminating against these mental health disorders in the student-athlete population. However, further research is necessary to confirm this.
Additionally, it has been noted in the literature that student-athletes often underreport symptoms of mental health.25 It is possible that the MHRS results in our study are an underrepresentation of the true symptoms student-athletes may be experiencing during the preparticipation physical evaluation. However, as our methods reflect the typical preparticipation mental health screening process in the collegiate setting, our results are likely an accurate representation and mirror the screening results that would occur within a collegiate setting. Despite these limitations, given that the MHRS was developed for alcohol screening and other validated measures appropriate for the athletic population exist, the athletic health care team (ie, ATs, team physicians, and licensed mental health professionals) should identify screening tools most appropriate for their patient population.
Future Research
Future researchers should seek to confirm our findings in other subpopulations, such as secondary school and professional athletes, as well as other collegiate athletic populations, including NCAA Divisions I and III, junior colleges, and the National Association of Intercollegiate Athletics institutions. The NATA recommends mental health screening at the secondary school level, and future researchers should focus on validating appropriate measures in this population, such as the PHQ-9 and GAD-7.6 Lastly, future researchers should continue to focus on validating mental health measures in the student-athlete population for other common mental health conditions such as disordered eating behaviors, insomnia, alcohol and substance use, and attention-deficit/hyperactivity disorder. As the MHRS has been recommended for use in 3 statements published in the Journal of Athletic Training, we recommend that the NATA Foundation Pronouncements Committee update these guiding documents to reflect best practices in screening using validated measures for mental health concerns.
CONCLUSIONS
This was the first study to investigate the clinical utility of the MHRS in a collegiate student-athlete population. The majority of participants in our study indicated yes to at least 1 item on the MHRS that warranted a referral to a mental health professional, with women having significantly more symptoms on the MHRS compared with men. Of 147 psychological evaluations, 33 participants screened positive for at least 1 mental health condition, with panic disorder and generalized anxiety disorder being the most common. The MHRS demonstrated high sensitivity but low specificity, resulting in a high false-positive rate. The overall accuracy of the MHRS was 40.14%, with an area under the curve score of 0.762, indicating fair discrimination capability. The optimal cutoff score for identifying mental health concerns was determined to be ≥4 on the MHRS; however, the sensitivity of 63.6% and specificity of 76.3% at this cutoff score is a continued area of concern. The data highlight the low clinical utility of the MHRS as a screening tool. Other validated tools, such as the PHQ-9 and GAD-7, are recommended.
FINANCIAL DISCLOSURE
Dr Keenan reports grants from the Eastern Athletic Trainers’ Association Research Grant during the conduct of the study and other from PROmotion Health, LLC, outside the submitted work.
Mental Health–Related Survey (MHRS) data.
Receiver operating characteristic curve for the Mental Health–Related Survey.
Contributor Notes