Capture of Time-Loss Overuse Soccer Injuries in the National Collegiate Athletic Association's Injury Surveillance System, 2005–2006 Through 2007–2008

Surveillance of Overuse Injuries

When overuse is a contributing factor, injuries can be particularly difficult to classify in injury-surveillance systems.⁶ Overuse injuries are traditionally described as injuries that result from repetitive stresses, progress over time, and do not have a distinct onset incident.^7,8 Due to their gradual onset and repetitive nature, overuse injuries can be challenging to identify, and consensus is currently lacking as to how to define and report overuse injuries in injury-surveillance systems.^9–12 Previous authors^13–15 reported that approximately half of sports injuries were the result of overuse. In a separate study,¹⁶ collegiate ATs who participated in injury surveillance indicated that they reported only 60% of the overuse injuries they treated, even though such injuries accounted for nearly 50% of the total injuries treated. Thus, a portion of the overuse injuries that affect athletes and medical staff are likely not represented in injury-surveillance data.

The purpose of our investigation was to estimate the capture rate of the NCAA ISS for time-loss medical-attention overuse injuries among men's and women's soccer teams. We also described the similarities and differences between how the NCAA ISS and medical records captured time-loss medical-attention overuse injuries and detail factors associated with disagreement between these sources regarding overuse as the mechanism of injury.

Data from the 2011 NCAA ISS validation study by Kucera et al¹⁷ were used to conduct these analyses. These data remain relevant as they represent time-loss medical-attention overuse injuries, which continue to be evaluated, treated, and recorded in current ISSs. Descriptive epidemiologic investigations of men's and women's soccer injuries using NCAA ISS data were published in 2007.^18,19 National Collegiate Athletic Association ISS data from 2004 through 2009 were also used in 2 investigations^20,21 of hamstrings injuries in soccer players. Additionally, these data have been cited in at least 35 publications that addressed a range of collegiate athletes' injuries.²² Given that other surveillance systems use the time-loss medical-attention injury definitions, it is important to study the representation of overuse injuries among limited time-loss medical-attention injuries in surveillance systems, and the 2011 validation study is the best source for data quantifying the completeness of time-loss medical-attention surveillance systems in capturing overuse injuries.

Parent Study

As indicated, the data used in this study came from a 2011 validation study¹⁷ that compared data from 664 injury events of the NCAA men's and women's soccer teams' medical records during the 2005–2006 through 2007–2008 seasons with those from the NCAA ISS injury records for that timeframe. Fifteen schools were included in the study, and the recruitment methods were previously published. Up to 3 years of injury records for each team were obtained. All NCAA ISS data for all consenting soccer athletes during the timeframe of the initial study were available for the parent study. Parallel medical records maintained by the university ATs, including hardcopy injury assessments, rehabilitation and progress notes, coaches reports, clinical notes from other clinicians (eg, physicians, physical therapists), and records from electronic databases other than the NCAA ISS, were considered the medical records source. The data were abstracted by 5 researchers, all with experience as collegiate ATs. Medical records were abstracted only for athletes who consented to participate. Efforts were made to reconcile misspellings of names and other discrepancies in the medical records. At the time of data collection, the NCAA ISS used the time-loss medical-attention definition. Extensive efforts were made to adhere to this definition regarding injuries in the medical records. See Kucera et al¹⁷ for further details regarding the selection and recruitment of schools and participants, data-collector training, and the data-abstraction process.

Overuse Injury Study

The current investigation was a secondary analysis of deidentified data from the parent study to examine the capture of time-loss medical-attention overuse injuries within injury surveillance and medical records. We used abstracted data from both the NCAA ISS and medical records regarding mechanism of injury, sex, division, presence of an undergraduate athletic training program, presence of a non-NCAA ISS electronic database, event details (injury date, activity, event type, event season), and injury details (diagnosis, body part, side of body, incident or recurrent, chronic), as well as notes from the abstractor's datasheet regarding the abstraction process, missing data, and quality of data. The current investigation was considered exempt from review by the University of North Carolina at Chapel Hill Institutional Review Board.

In the absence of a criterion standard, capture-recapture analyses have been successfully used to estimate the incidence of specific outcomes in populations from various reporting sources.²³ The purpose of capture-recapture analyses has been to estimate both the total occurrence of an outcome or condition and the capture rates of individual sources. We focused on describing the capture rates of the NCAA ISS, medical records, and both sources for injuries with an overuse mechanism. To be included in these analyses, an injury event required that overuse be assigned as the mechanism of injury in 1 or both sources. The assignment of an overuse/gradual-onset mechanism of injury was determined by the ATs for the NCAA ISS. In the medical records, the mechanism of injury for individual events was classified as overuse by the data abstractor reviewing the medical records. The capture-recapture analysis was used to describe the variability between the NCAA ISS and medical records for capturing overuse injury mechanisms individually rather than to predict the total number of overuse injury mechanisms in this population. The capture rates of overuse injury mechanisms in the NCAA ISS, medical records, and both sources combined were estimated, as were the estimated number of overuse injury mechanisms not captured by either source. Estimates of overuse mechanisms of injury were also calculated within strata of covariates including sex (male or female), NCAA division (I, II, or III), presence of an undergraduate athletic training program (yes or no), and use of a non-NCAA ISS electronic medical record (yes or no).

Data Analysis

Hook and Regal²³ presented the formula for estimating the potential overlap of coverage from the 2 sources using a 2 × 2 table where a, b, c, and x are defined as follows: a = overuse mechanisms of injury identified in both the NCAA ISS and medical records, b = overuse mechanisms of injury identified in the NCAA ISS but not the medical records, c = overuse mechanisms of injury identified in the medical records but not the NCAA ISS, and x = overuse mechanisms of injury not identified in either source (or x = bc/a). With this estimation of x, the total number of overuse mechanisms of injury can be estimated as N = a + b + c + x. From the estimation of the total N (total overuse mechanisms of injury), the capture rates for the NCAA ISS, medical records, and combined NCAA ISS and medical records can be estimated. For example, the capture rate for the NCAA ISS can be calculated by (a + b)/N.^17,23

Kappa analyses were calculated to compare the level of agreement among covariates; however, these calculations were limited to records that appeared in both sources. Kappa analyses were conducted to obtain an estimate of agreement that also considers agreement according to chance. Strength of agreement was adapted from Landis and Koch,²⁴ where κ < 0 = poor agreement, 0%–20% = slight agreement, 21%–40% = fair agreement, 41%–60% = moderate agreement, 61%–80% = substantial agreement, and 81%–100% = almost perfect agreement. Effective agreement²⁵ was also estimated as the percentage agreement for the following covariates: activity at time of injury, event type, incident or recurrent, body part injured, side injured, injury type, outcome, and injury severity. Effective agreement was calculated as the percentage of NCAA ISS and medical records that agreed on the value for that covariate.

The distribution of the characteristics of the injury events was calculated (n and %) for strata in which the mechanism was overuse in both systems and in which the mechanism was overuse in 1 system. We calculated the distributions for the NCAA ISS and medical records separately within each group. The primary author (K.R.) reviewed all data-abstractor notes and recorded inconsistencies between the NCAA ISS and medical records (eg, differences in time-loss estimates between the sources), missing data in either system, missing data in the abstractor records, and any notations the abstractor recorded. The analysis of data-abstractor notes was used to help us understand the difference between systems and to add context to the results.

Capture of Time-Loss Overuse Injuries

From the parent study, 64 records described the mechanism of injury as overuse in 1 or both sources, ie, the NCAA ISS, the medical records, or both (Figure). Of these 64 records, 48 events had an overuse injury mechanism in the NCAA ISS, and 44 had an overuse injury mechanism in the medical records. Overlapping events that were captured as overuse in both sources totalled 28. Using capture-recapture analyses, an estimated 11.4 events were missed (or not captured as overuse) by either source. Therefore, the number of time-loss overuse mechanisms of injury in this sample of n = 664 total injuries was estimated to be 75.4, or 11% of all time-loss medical-attention injuries (Table 1).

View Figure

• Download as Powerpoint
• Download Figure

Table 1.  Capture-Recapture Analysis for Injuries With an Overuse Mechanism in 1 or Both Sources

View Fullscreen

Overall, the capture rates for time-loss medical-attention injuries with an overuse mechanism were 63.7% for the NCAA ISS and 58.4% for the medical records.

Capture rates varied by institutional and descriptive characteristics. The most notable variation was the lower capture rate of overuse mechanism of injury for women's soccer compared with men's soccer (χ²: 9.60, P = .002 for ISS capture, and χ²: 6.45, P = .011 for capture by NCAA ISS and medical records). Additionally, capture of overuse mechanisms of injury tended to be better in schools with an undergraduate athletic training program, specifically in the capture rate of medical records, although the difference was not statistically significant (χ²: 3.19, P = .074).

Agreement

Ten events with an overuse mechanism were reported to the NCAA ISS but were not found in the medical records. These 10 events were not included in either the κ or effective agreement calculations and resulted in an altered sample size for the total number of records that appeared in both sources (n = 54).

Injury details (incident or recurrent, body part, side of body) had the highest κ agreement (78.7%–88.8%), whereas the event (activity = 53.9% and event type = 73.6%) and return-to-play (outcome = 60.7% and severity = 76.8%) details had the lowest κ agreement (Table 2).

Table 2.  Kappa and Effective Agreement Between National Collegiate Athletic Association Injury Surveillance System and Medical Records for Event, Injury, and Return-to-Play Details

View Fullscreen

Effective agreement was higher overall compared with κ agreement, except for injury severity (Table 2). Injury details also had the highest effective agreement, although event type (an event category) and injury outcome (a return-to-play category) had higher effective agreement than injury type (an injury-detail category).

Disagreements between the NCAA ISS and medical records were often in either close proximity (eg, discrepancies between the sources regarding body part were anatomically close, such as hip versus thigh) or temporality (eg, close to half of all discrepancies between the sources regarding number of days lost from sport participation were less than 3 days apart; Table 3).

Table 3.  Characteristics of Disagreements Between the National Collegiate Athletic Association Injury Surveillance System and Medical Records for Injuries Classified as Overuse in Either Data Source (n = 52^a)

View Fullscreen

Characteristics of Time-Loss Overuse Mechanism of Injury Events

Nearly all overuse mechanisms of injury affected the lower extremity. Among time-loss medical-attention overuse injuries in which overuse was the mechanism in both systems, most injuries were reported to occur in the lower leg (NCAA ISS = 35.7%, medical records = 32.1%; the body part injured disagreed for 1 of the 28 events). Among the records in which overuse was the mechanism in only 1 source, the majority of the injuries were to the hip/thigh (NCAA ISS = 26.9%, medical records = 34.6%). Overall, a larger percentage of data were missing for the injury characteristics in the medical records than in the NCAA ISS. For most variables, the medical records had 2 to 5 times the number of not specified or don't know responses per injury than did the NCAA ISS.

Capture-Recapture Analysis

The NCAA ISS captured 64% of time-loss medical-attention overuse mechanisms of injury in this population, which is considerably less than the 88% capture rate of the NCAA ISS for total time-loss medical-attention injuries in the parent study.¹⁷ The capture rate for overuse mechanisms of injury was even lower if the data source used was only the medical records (58.4%) rather than the NCAA ISS (63.7%). The higher capture rate in the NCAA ISS as compared with the medical records may be related to the format of the electronic medical records. In the NCAA ISS, the AT chooses a mechanism of injury from a list of provided options (contact with player/competitor, contact with playing surface, contact with playing apparatus, contact with out of bounds objects, acute noncontact, overuse/gradual onset, illness, infection, other, and don't know). Although other offers a write-in option, most ATs reported mechanisms in 1 of the identified categories.^3,4 The comparison medical records included data from a variety of sources and did not require the AT to commit to a specific mechanism of injury. Although the NCAA ISS had a better capture rate, it is clear that time-loss medical-attention overuse mechanisms of injury were not well captured by either system.

Additionally, it should be noted that, despite the NCAA ISS and medical records capturing approximately similar percentages of overuse injury mechanisms, they captured different individual events. Only 37.1% of included injuries were captured as overuse mechanisms of injury in both the NCAA ISS and medical records. The variability in these results regarding the reporting and classification of overuse mechanisms of injury in both sources is likely related to several factors, including the lack of consistency regarding the definition of overuse in surveillance systems specifically and the literature in general, differences in purpose and function between surveillance systems and medical records, and the nature and onset of the overuse injuries themselves.^11,17,26

The causes of these injuries, which result from repetitive stress and progress over time, likely contributed to the variability in the records as well.^7,8 Because an athlete can seek medical attention at any point in the injury process, variable symptoms may be present during the initial evaluation and classification of the injury. This may result in an overuse injury being misidentified or misclassified due to the complexity and timing of the assessment. For example, if a patient with an overuse injury presents only after it has progressed into a severe or inflamed injury, it may be assessed incorrectly, documented as an acute injury, or both.²⁷

The differences in purpose and function of injury-surveillance systems and medical records also likely contributed to the differences in capture rates for overuse mechanisms of injury. Medical records document relevant information about the individual injury and treatment provided for the athlete.²⁸ In addition, medical records provide continuity of care and a thorough history of the injury, treatment, and rehabilitation. Such records are also used for billing and other administrative purposes. Surveillance systems collect data on a population level and (for systems such as the NCAA ISS) ideally include information on individuals with and without the outcomes of interest.² The scope of the data collected by the surveillance system is defined according to the outcomes of interest and purpose of the injury surveillance.^1,29 The different purposes of these systems may contribute to the different capture rates of overuse mechanisms of injury.

Also, although ATs who participated in the NCAA ISS were trained on how to use the online system and enter data into the system, the diagnosis and assessment of injury, including mechanism of injury, were ultimately left to the AT to determine.³⁰ Providing operational definitions for the outcomes (eg, overuse injuries) and variables of interest (eg, mechanism of injury) for use by surveillance data collectors would likely improve the reporting consistency for overuse mechanisms of injury and overuse injuries as a whole in the future.

Effect of Sex

The different capture rates between men's and women's soccer may be related to the fewer time-loss medical-attention overuse mechanisms of injury reported among female soccer athletes (29 among women versus 35 among men). These results are inconsistent with the literature,^16,31,32 in which overuse injuries have been reported at higher rates among female athletes compared with males in previous studies of collegiate athletes. The higher capture rate in men's soccer may be due to sex differences in reporting injuries to ATs. Female athletes may report injuries more quickly after onset than male athletes, which can result in misclassification of injury mechanisms due to the timing of when care is sought.³³ If a female athlete reports an injury before it results in significant limitation from sport, she may receive treatment before the injury meets the time-loss requirement for entry into the NCAA ISS, which may depress the capture rate of overuse injuries.³⁴ Continued research into sex differences in the reporting of both overuse injuries and injuries overall may help distinguish differences in the processes for capturing overuse injuries and allow for targeted modifications to existing systems to better capture these injuries across all athletes.

Agreement

The agreement (κ) between the NCAA ISS and medical records was highest for the injury details (eg, incident or recurrent, body part injured, side injured) and was consistent with results from the previous validation study.¹⁷ Injury details, such as incident or recurrent, body part injured, side injured, are usually documented in the medical record and easily discerned by the AT. Conversely, event and return-to-play variables tended to have lower agreement, and coding of these variables may require greater judgment from ATs to classify. Injury severity was derived from the number of days lost from sport participation and had the lowest agreement as assessed by κ. This disagreement may be due to the increased opportunity for disagreement between the NCAA ISS and medical records, as severity depended on accurately recording the date of full return to participation.

Training data collectors generally involves specific education on the data-collection and -entry process, rather than on clinical skills or clinical judgment.³⁰ Therefore, the individual AT's assessment habits and skills may influence the data that are entered into the system. Also, any change in the athlete's presentation between the times of the NCAA ISS and medical record documentation will be reflected in the lack of agreement between the 2 systems.

Injury Characteristics

The majority of overuse injury mechanisms in this dataset were to the lower extremities, a finding that is consistent with the literature^16,18,19 on both overuse injuries as a whole and soccer injuries specifically. Disagreements between the NCAA ISS and medical records classifications for body part were anatomically adjacent, in that 1 system would document an ankle sprain, and the other would document a lower leg injury. Disagreement could be due to a transcription error while reporting the injury to the NCAA ISS or medical records or a change in the injury assessment or diagnosis between the time of NCAA ISS and medical record reporting. Another possibility is that only 1 record was updated as an injury progressed. We did not assess the documentation timelines or changes to documentation in this investigation, but that information would be a worthy inclusion for any future validation study of overuse injuries in surveillance systems.

The variable of injury outcome demonstrated inconsistencies between the NCAA ISS and medical records in statistical agreement (Table 2). Disagreements may be due to the nature of overuse injuries: athletes commonly limit participation rather than missing time from play altogether.^16,35 The progression of overuse injuries can lead to variations in return-to-play dates and discrepancies between surveillance and medical records (eg, coaches reports, progress notes, daily treatment logs, SOAP [subjective, objective, assessment, plan] notes), underscoring the importance of clear definitions and protocols for the ISS to improve consistency within injury-surveillance data.

Recommendations for Surveillance of Overuse Injuries

The substantial clinical variability we observed clearly speaks to the need for better training and standardization regarding what constitutes an overuse injury. Such training would be valuable for students in academic athletic training programs as well as for participants in surveillance systems. Thus, a process should be developed for the formulation and implementation of standardized definitions for the more frequent overuse injuries reported to surveillance systems. Standardized definitions ideally should be integrated into clinical education and function diagnostically for ATs; they can also serve as useful guidelines for injury-surveillance purposes. The formulation and implementation of a standardized definition for overuse injuries have the potential to improve both clinical care for these injuries and their representation in injury data and the literature. Although several methods for injury classification have been proposed, none are currently accepted as a criterion standard.^29,34,36,37 Such a standard would be immensely beneficial in the case of difficult-to-classify injuries, such as overuse.

We also recommend that future researchers of overuse injuries use multiple data sources. Notably, a large number of data were missing regarding the characteristics of the injury in the medical records (8.6% for mechanism) as compared with the NCAA ISS. On the other hand, the course of treatment and management of the injury may be critical to identify overuse injury, and this information is typically available only from clinical records. Data abstractors often noted the presence of extensive information in clinical records regarding 1 variable (eg, time lost from sport derived from the date of full return to participation), which may be important to identifying the specific causation behind a single overuse injury, but the variable that contained those details and the specific information provided in individual records varied. This agrees with our findings that neither source was complete individually for time-loss overuse mechanisms of injury, suggesting that both sources should be used for complete capture of overuse events in epidemiologic investigations. This may not be practical for daily injury-surveillance purposes, yet it may be critical to specific investigations into overuse mechanisms of injury. Further, a chart review may offer more contextual information that could also complement the epidemiologic findings. Finally, new methods of data collection, including athlete self-reported injury data, are under development and in use in prospective studies. Such advancements would allow athletes' data on the symptoms and consequences to be obtained and incorporated into future multivariate studies.^9,38

Limitations

This investigation used data from a prior study⁴ that was designed to assess all injuries, not overuse injuries or overuse mechanisms of injury specifically. Therefore, the original population was limited to 664 injuries, and our resultant sample size with the overuse mechanisms of injury was small. Additionally, the injury inclusion criteria were limited to a time-loss medical-attention injury definition and to soccer athletes. As a result, almost all injuries were to the lower extremity (which accounted for 80% of the total overuse mechanisms of injury in soccer players).²¹ The restriction to time-loss medical-attention injuries was necessary considering that the data originated from the NCAA ISS, which used this injury definition until 2009.

The study results might have differed if non–time-loss injuries or overhead athletes (such as swimmers or baseball or softball athletes) had been included. The current findings were also limited by the available validation data, which have been collected only for the NCAA ISS. Although this system has evolved since these data were collected, the current NCAA Injury Surveillance Program was based on the NCAA ISS platform and continues to collect data on time-loss injuries. Replicating this study with all medical-attention injuries and in a wider variety of sports would provide additional insights into the capture of overuse mechanisms of injury by injury surveillance.