Question Format Matters: Do Athletes Really Know the Signs and Symptoms of a Sport-Related Concussion?

Sport-related concussion (SRC) has been studied extensively in the past decade, and various professional statements^1–3 have emphasized the importance of elevating athlete awareness and knowledge about SRC through education. Authors of previous studies of SRC knowledge and education intervention effectiveness have commonly used a multiselect checklist item to assess an athlete’s ability to recognize true signs and symptoms (S&S) among incorrect distractor options.^4–6 Although this method may identify the athlete’s ability to recognize SRC S&S from a list, an open-ended recall question format may be more suited to identifying the ability to retrieve appropriate SRC S&S from long-term memory (ie, existing knowledge).⁷Recognition is one’s ability to identify subject matter through the process of familiarity, whereas retrieval of knowledge or recall refers to one’s ability to search the relevant information in long-term memory and is influenced by the extent to which the person was exposed to the information previously.⁸ Consequently, earlier investigators who focused on the recognition of SRC S&S may not have comprehensively assessed athletes’ true understanding.

Kerr et al⁹ examined the knowledge of SRC S&S among parents of US middle school–aged children (10 to 15 years old) using a multiselect item and an open-ended question format. Parents were able to identify more SRC S&S when asked via the multiselect item, which may indicate that the question format can influence the S&S knowledge score. Furthermore, the patterns and frequencies of answers were also different between the 2 assessment methods; the top 3 answers via the multiselect approach were dizziness (90.2%), blurred vision (87.4%), and balance problems (86.4%), whereas in the open-ended question format, they were headache (49.5%), dizziness (44.4%), and nausea or vomiting (28.0%).⁹ If the inflation from the multiselect question format is also observed in athletes, it could potentially influence our degree of confidence in an athlete’s ability to identify and self-report an SRC. However, to our knowledge, no researchers have directly addressed the influence of question type in SRC S&S assessment among athletes.

In addition, most studies with results that supported the SRC consensus statement¹ were conducted in North America and Europe. Exploration of the understanding of SRC among athletes in Asia, which has much less organizational and societal exposure to SRC information, is urgently needed.¹⁰ Therefore, the primary aim of our study was to examine if differences existed in SRC S&S answers between 2 modes of assessment among collegiate athletes in Japan. We also assessed whether previous exposure to SRC education and a history of SRC influenced athletes’ SRC S&S knowledge.

METHODS

We conducted a cross-sectional survey to examine the SRC knowledge of collegiate lacrosse athletes in Japan. An online survey (Qualtrics) was sent to registered members of the Japan Lacrosse Association for the 2021–2022 academic year (n = 8530) and was open from August 2021 through September 2021. Survey questions were adapted from Beidler et al¹¹ with modifications in answer options to match the cultural nuances of the Japanese collegiate sports setting (eg, modifications in the list of sports played to include domestic sports, changing examples of licensed medical professionals to fit the Japanese medical system). The survey was first translated into Japanese by 2 content experts (Y.H., M.O.). It was then reviewed by 3 certified athletic trainers who were bilingual in Japanese and English for item clarity. The translated survey was tested on Japanese college students (n = 11) to verify its clarity and that the intent of the questions was delivered accurately.

We first asked an open-ended question that required the athletes to list SRC S&S so that we could evaluate their knowledge under free-recall conditions. The next item measured athletes’ level of understanding of common SRC S&S using a list of 34 options,⁴ in which 20 were true S&S (eg, confusion, headache, amnesia) and 14 were incorrect distractors (eg, fever, joint stiffness, bleeding from the nose). Athletes were asked to select all S&S that they believed were associated with SRC and were not informed about the presence or number of false options on the list. The survey format restricted athletes from changing answers in the open-ended questions once they proceeded to the multiselect item. In the same survey, previous exposure to SRC education (yes [EDU_YES], no, or I do not know [EDU_NO]) and previous history of diagnosed SRC (yes [HX_YES], no [HX_NO]) were queried using multiple-choice questions.

The open-ended answers were first coded using text-mining software (KH Coder 3) to identify clusters of similar terms. Clusters of terms representing similar concepts were then merged to form a category (ie, SRC S&S). To allow comparisons against the multiselect answers, the list of 34 S&S served as the main category names when open-ended answers fit in them. Open-ended answers that fell outside of the 34 S&S categories were labeled a new category and counted separately. Open-ended answers that fell outside of the main category were assessed by Y.H. and M.O. to determine whether the reported S&S were correct or incorrect.

All data were analyzed using SPSS (version 27; IBM Corp). The number and proportions of responses by participants were calculated using descriptive statistics. Pearson correlation was used to analyze the relationship between scores from the 2 question formats; to prevent inflation in the number of correct answers that only appeared in open-ended answers, we counted open-ended answers that fell within the 20 correct S&S categories for this comparison. Unpaired-samples t tests were used to (1) compare the means of the numbers of correct and incorrect answers in the list format between participants who responded I do not know and participants who responded with at least 1 sign or symptom in the open-ended question and (2) compare the means of correct answers by previous SRC education and history of diagnosed SRC. Furthermore, in all S&S with at least 5 responses in the open-ended question, we compared the proportion of correct answers by question type per symptom and calculated its odds ratio using the McNemar test. Mean difference (MD), 95% CIs, and Cohen d for effect sizes were calculated where appropriate. Statistical significance was set a priori at <.05. This study was approved by the Human Research Ethics committee of Waseda University (#2021-095).

RESULTS

The response rate of the survey was 35.9% (n = 3065), with 71.9% (n = 2203; of these, 548 [24.9%] answered I do not know) reporting no previous exposure to SRC education (EDU_NO) and 92.3% (n = 2829) reporting no history of diagnosed SRC (HX_NO). Scores from the open-ended question and multiselect formats were moderately correlated (r = 0.34, 95% CI = 0.31, 0.37; P < .001).

Open-Ended Question

For the open-ended question, 4.5% (n = 137) of athletes responded I do not know. After these respondents were excluded, the average number of correct true S&S reported was 1.6 ± 1.2 (minimum = 0, maximum = 10, median = 1). Only 15 (0.5%) participants reported at least 1 false distractor that corresponded to 1 of those in the multiselect item. The top 5 true SRC S&S open-ended responses were headache (34.1%, n = 999), nausea or vomiting (33.0%, n= 966), loss of consciousness (28.4%, n = 832), dizziness (28.3%, n = 828), and memory loss (15.7%, n = 459; Table, Figure). Apart from the 34 S&S included in the multiselect options, the top S&S open-ended responses were collapse or inability to get up (true; 8.3%, n = 242), muscle spasm (false; 4.4%, n = 129), slurred speech (true; 3.8%, n = 110), and lightheadedness (true; 3.6%, n = 106).

Table Sport-Related Concussion Signs and Symptoms Reported by Japanese Collegiate Lacrosse Athletes (N = 3065) by Question Format Type

View Fullscreen

View Figure

• Download as Powerpoint
• Download Figure

DISCUSSION

In this study, we are the first to investigate question-type differences in SRC S&S knowledge in collegiate athletes. Our findings are consistent with those of Kerr et al⁹ among a sample of US parents of middle school children. The collective results of these 2 studies indicate that an open-ended item may more accurately assess an individual’s true ability to retrieve previously learned SRC S&S information. Though the effect was minimal, having received SRC education previously or having a history of ≥1 diagnosed SRCs may result in more robust S&S knowledge. These findings could have major implications for both SRC awareness and knowledge assessment as well as clinical applicability when considering athletes’ ability to readily recognize S&S when they or a teammate sustain an SRC during sport participation.

Open-ended question and multiselect formats displayed considerable differences in the number of correct and incorrect SRC S&S reported, with a greater number of the true S&S recognized using the multiselect format (Table, Figure). This is not surprising, as recognition-question formats are well known to result in increased performance compared with free-recall items.⁷ However, we also observed that the selection of incorrect distractor options increased from 0.5% to 60.4% with the multiselect format (Table). Taken together, these findings provide evidence that athletes remain confused regarding SRC S&S, athletes may be guessing when faced with multiselect items, or both. Therefore, accurate knowledge of SRC S&S may be best measured by asking an open-ended question or by also considering the identification of incorrect distractor items in the total knowledge score (eg, correctly recognized true items − incorrectly identified distractor items = true knowledge). Furthermore, it was also concerning that >130 respondents answered the open-ended item by stating that they simply did not know any SRC S&S. Although these individuals were able to identify some SRC S&S using the multiselect format, their recognition of these answers does not translate to the ability to freely recall the information from long-term memory. These results warrant a reevaluation of the utility of the previous literature in which authors found acceptable SRC S&S knowledge in athletes using the multiselect item format. Clinically, this gap emphasizes the importance of using both approaches to conduct a multifaceted evaluation of SRC S&S, such as asking injured athletes to describe how they are feeling during their immediate on-field assessment (ie, open ended) and using the symptom checklist included in the Sports Concussion Assessment Tool (SCAT) during the SRC evaluation (ie, multiselect).¹

When the correct SRC S&S answer list was compared between the multiselect and open-ended formats, the top 10 answers were similar except for confusion (multiselect: eighth popular correct answer, 45.1% [n = 1382]; open ended: 14th popular correct answer, 0.2% [n = 6]). The proportions of answers were vastly different, yet dizziness, feeling off balance, headache, memory loss, blurred vision, loss of consciousness, nausea or vomiting, feeling slowed down, and fatigue or low energy were present in both question formats, which covers the following symptom clusters developed by Harmon et al²: ocular, vestibular, cognitive, fatigue, and headache-migraine. This leaves anxiety-mood–related SRC S&S as the least recognized symptom cluster in both question formats among our participants, which agrees with previous examinations of collegiate athletes.^12,13 Although anxiety-mood–related S&S are likely to appear in combination with other S&S,¹⁴ future athlete education on SRC S&S should emphasize this symptom cluster to raise awareness and normalize the presence of psychological challenges after SRC.

When compared by previous exposure to SRC education, the MD in the number of correct answers was statistically different, but the magnitude of clinical relevance was questionable. This outcome may have reflected the cultural differences in the quality and frequency of SRC education in Japan compared with other countries. It was not until 2019 that evidence-based SRC information became readily available in the Japanese language.¹⁵ Therefore, despite the reporting by some participants of EDU_YES, the extent and quality of the actual SRC education they received is unknown. The comparison by a history of diagnosed SRC also resulted in a statistical difference in the MD with trivial clinical relevance. Earlier researchers also failed to demonstrate a strong relationship between a history of SRC and current SRC knowledge,^16,17 so sustaining an SRC may not independently improve one’s SRC knowledge despite the statistical significance we observed. Thus, investigators and health care providers should not assume that athletes who have received SRC education or sustained an SRC previously are more knowledgeable about SRC S&S based on these experiences alone.

CONCLUSIONS

The open-ended and multiselect-question formats resulted in different SRC S&S knowledge assessments, with the possibility of inflation in the SRC recognition results via the multiselect-question format. Researchers and clinicians should acknowledge the differences between recognition and free recall when assessing one’s understanding of SRC and managing athletes with suspected SRC.