Reply
Dear Editor:
I appreciate the opportunity to respond to the concerns raised by Mr Beachy in his letter to the editor regarding our recently published article, “The first decade of Web-based sports injury surveillance: descriptive epidemiology of injuries in United States high school football (2005–2006 through 2013–2014) and National Collegiate Athletic Association football (2004–2005 through 2013–2014).”1 Although more than 100 peer-reviewed papers reporting data captured by the High School Reporting Information Online (HS RIO) surveillance study have been published to date, many in the Journal of Athletic Training, journal-mandated word counts rarely provide the opportunity to fully describe the HS RIO methods or the extensive steps that are taken to ensure that the captured data are of the highest possible quality. Evaluating both the internal and external validity of the captured data is an arduous task for those administering large surveillance systems, but, as Mr Beachy asserted, it is my duty as the primary investigator of HS RIO. From its establishment in 2005, I have been transparent regarding my commitment to this effort in annual summary reports that contain both methodologic details and reporter compliance information. These reports are available on the HS RIO Web site: http://www.ucdenver.edu/academics/colleges/PublicHealth/research/ResearchProjects/piper/projects/RIO/Pages/Study-Reports.aspx.
Before I address the question of external validity, I must first note Mr Beachy's use of injury counts to compare data across studies. A more appropriate metric, when available, for comparing injury data across studies, populations, or population subgroups is injury rate, which is calculated as the injury count divided by a population-based exposure denominator. High schools in the United States vary dramatically from small schools playing 8-man football with rosters ≤15 to large schools with 3 squads and roster totals ≥100. We would not expect the same number of time-loss (TL) injuries to occur in schools with very small versus very large team rosters. Thus, rates, which adjust for team sizes and numbers of athlete-exposures, should be used for comparisons when available. That said, solely for the purpose of responding to Mr Beachy, I will use injury count data to further demonstrate why his concern is unwarranted.
A study's external validity is most often evaluated by comparing the data to the known literature. As Mr Beachy noted, because the data from his school are unpublished, I have not previously compared them with the HS RIO data. It is not surprising, however, that Mr Beachy observed a higher count of TL injuries at Punahou School. The HS RIO provides aggregate data captured from a large national sample of US high schools. As demonstrated in the Table, the number of injuries reported per participating school varies widely. Although Mr. Beachy's self-reported 73.6 TL injuries would put his school in the upper 25th percentile of schools participating in HS RIO during 2013–2014, his injury count is well below that of the school reporting the highest injury count. However, Mr Beachy's reported injury count is higher than the data presented in the 4 articles he referenced. Should the accuracy and validity of Mr Beachy's data be called into question because they are well below the data of at least 1 HS RIO school or because they are well above data published in several articles? Obviously, I do not believe so given the expected variations across schools.
Because injury rates and patterns can vary over time, it is also not ideal to evaluate the external validity of HS RIO by comparing it with articles published more than 3 decades ago, as Mr Beachy did. The most appropriate way to evaluate the external validity of HS RIO is to compare its data with those of the National Athletic Trainers' Association's (NATA) National Athletic Treatment, Injuries and Outcomes Network (NATION),2 currently the only other large national high school sports injury-surveillance system in the United States. As Mr Beachy noted, the HS RIO data were relatively consistent with the NATA NATION data.
Internal validity is much more easily evaluated, and I have annually used several approaches to promote high-quality data reporting and to subsequently evaluate the internal validity of the HS RIO data. I will outline a few here. Because I strongly believe that only trained sports medicine clinicians can consistently provide high-quality data to an injury-surveillance system over time, only schools with an NATA-affiliated certified athletic trainer are eligible to participate in HS RIO. Athletic trainers at participating schools have options to receive self-paced slide or telephone training on the reporting methods and, as a reminder, the definitions of injury and athlete-exposures are provided at the top of each report form along with a brief tutorial on how to calculate and report the latter. My Internet-based data-reporting tool uses integrated skip patterns to reduce reporter fatigue, includes several real-time logic checks to notify reporters of entries that may be out of the expected ranges, and provides a color-coded feedback tool to alert reporters of incomplete data. The HS RIO staff conducts routine audits to ensure that each athletic trainer provides weekly data and that the reported data meet quality standards. When staff members identify potential concerns, they e-mail the athletic trainer, noting the potential data problem and offering assistance. At the end of each academic year, each school's data undergo a quality check during the closeout process, and schools not meeting data quality standards are removed from the final aggregate sample. The data set aggregated from the final sample of high schools is then subjected to another data-quality check, and data cleaning and database management are conducted by HS RIO staff. A 5% random sample of schools is asked to participate in an internal validity check. These schools supply a copy of their health log (after personal athlete identifiers are removed) for 2 weeks of the academic year, randomly sampled from all weeks in which they reported at least 2 injuries. Study staff compare the provided health logs with the data reported to HS RIO and calculate sensitivity, specificity, and positive and negative predictive values. Reporter-compliance metrics are included in the annual report.
Yet one more step is taken to ensure that publications reporting HS RIO data are accurate. My data-use agreement requires that before HS RIO data are released in any public forum (eg, paper, abstract, poster, presentation), the research team must submit their work to me so my staff can replicate all data presented and confirm that there were no errors in data analyses or interpretation. I believe, considered together, the aforementioned steps demonstrate how seriously I take my responsibility to “verify the accuracy and validity of the submitted data before they are made available to the athletic community.”
Over the past 14 years, as I have published articles reporting HS RIO data, I have frequently received similar concerns raised by reviewers who are less familiar with HS RIO. The concerns can be summarized in 2 questions. First, “The data in this paper differ from what I personally experience in my clinic, so how can they be true?” Mr Beachy stated that his observation of more TL injuries at his school than reported by HS RIO “should raise questions about the accuracy of the reported data.” Second, “How can athletic trainers be trusted to accurately report clinical data on sports injuries?” Mr Beachy suggested that “individuals collecting and reporting the data are often the weak link.” Although I have come to expect these questions from some physicians in clinical specialties who are not always knowledgeable about the full spectrum of sports injuries and athlete health that athletic trainers encounter, in my experience, the vast majority of athletic trainers meet very high standards, not only in the clinical care they provide but also while participating in surveillance and research efforts.