The Influence of Education and Environment on Patient-Reported Outcome Use in Athletic Training Clinical Practice

Participants

Electronic surveys were distributed in April of 2019 to a unique, purposeful sample of 4000 individuals believed to be clinically practicing ATs at either the collegiate or secondary level. This number was chosen a priori to ensure a large and diverse sample of responses from clinically practicing ATs. Based on previously published survey-based research in athletic training^10,11,13 it was estimated that a minimum response rate of approximately 10% was feasible, which would give a minimum sample of 400 participants.

Email addresses from collegiate athletic department websites were culled using a randomized list of NCAA member institutions at the Division I, II, or III levels (N = 2000 emails). The list was developed by copying NCAA member lists from each division²³ into a spreadsheet. Random numbers were then generated for each row of the spreadsheet, and the spreadsheet was then sorted numerically based on the generated random numbers. Athletic department websites were visited in the order of the composed list until a list of 2000 emails was generated. The same process was completed using a list of JUCO member institutions (N = 250 emails) and a list of NAIA member institutions (N = 250 emails). The final 1500 emails for secondary school ATs were identified from the NATA Athletic Training Locations and Services Database (ATLAS), maintained by the Korey Stringer Institute at the University of Connecticut.²⁴ The number of ATs identified per state were weighted based on US Census population data, such that a total of 1500 emails were identified. For each state, the state map was selected and filtered to identify secondary schools with part-time or full-time ATs. Secondary schools were then clicked on at random until the appropriate number of unique emails were obtained for a given state.

Survey Development

The developed survey consisted of 3 main parts, which were presented in the following order: (1) Attitudes, Beliefs, and Practices relating to PROMs (23 items); (2) Access to Evidence-Based Practice (EBP) Information (5 items); and (3) Demographics and Educational Background (8 items) (see Supplemental Figure, available online at www.nataej.org).

Survey Distribution

Using commercially available survey software (Qualtrics XM, Qualtrics LLC), we distributed survey invitations via email with an anonymous link to the survey instrument and a requested completion date of 4 weeks after receipt. For surveys that were not completed in the first 2 weeks, 2 additional reminders requesting completion were sent approximately 1 week apart. Emails were scheduled such that initial invites and reminders all were sent on varying weekdays to optimize response rate. Email addresses that the system identified as “undeliverable” were replaced by additional email addresses from the same classification (NCAA, NAIA, JUCO, or secondary school) as the original addresses, using the same methods of random selection previously presented.

Data Analysis

Descriptive statistics (means/medians, standard deviations/ranges, and frequency counts) were calculated. Because of nonnormative distributions of responses (as indicated by Shapiro-Wilks tests), Kruskal-Wallis tests were used to examine attitude scores based on clinical setting, educational background (highest level of degree achieved, type of professional program completed), and access to scientific journals beyond those affiliated with the NATA. Pairwise comparisons were only examined if the overall test statistic was significant (P ≤ .05). Chi-square tests were used to compare bivariate responses regarding use of PROMS, with z-tests for independent proportions for post hoc comparisons of significant results (P ≤ .05), and where applicable, odds ratios with 95% confidence intervals (CIs) were calculated. Chi-square tests were also used to compare proportions of respondents with access to scientific journals based on traditional clinical setting.

Attitudes, Beliefs, and Practices Related to PROMs Use

Each participant who started the Attitudes and Beliefs section of the survey completed all 20 Likert-type items; therefore, all available data (N = 507) were used to calculate the Acquiring and Using subscale, Benefits subscale, and Total Attitude scores. Scores for the overall sample and by degree level/type, access to scientific journals, and year of graduation are presented in Table 2. Overall, respondents viewed PROMs positively, with median subscale scores and Total Attitude scores exceeding 3.0. There was no effect for highest degree level for Acquiring and Using (P = .168) or Total Attitude (P = .110). There was a main effect for highest degree level for the Benefits subscale (P = .029), with those having completed a clinical doctorate demonstrating a significantly more positive attitude toward the benefits of using PROMs when compared to individuals with a bachelor's degree (P = .032) or a master's degree (P < .001). Within degree types, a main effect for bachelor's degree type was observed for the Benefits subscale (P = .001), with individuals who completed curriculum-based bachelor's programs reporting more positive attitudes toward the benefits of using PROMs when compared to individuals who completed an internship-based bachelor's program (P = .002).

Table 2 Attitudes and Beliefs Regarding the Use of Patient-Reported Outcome Measures (Rating Scale 1–5, with 5 Representing Most Positive Views)

View Fullscreen

Among those with a master's degree, there were main effects for Acquiring and Using (P = .002) and Total Attitude (P = .030), but not for Benefits (P = .262). Those with athletic training–related postprofessional master's degrees reported more positive attitudes toward Acquiring and Using PROMs (P = .021) and overall Total Attitudes toward PROMs (P = .035) than did those with a non–athletic training–related master's degree. There were no differences on any of the attitude and belief scales based on employment setting (P = .821 to .878). Similarly, there were no differences on any of the attitude and belief scales between those completing their professional degree before 2014 and those completing their professional degree in 2014 or after (P = .098 to .137). Finally, those reporting access to scientific journals beyond those associated with the NATA reported significantly higher scores on all 3 scales: Acquiring and Using (P = .002), Benefits (P = .036), and Total Attitudes (P = .002).

Use of PROMs

Overall, 60% (n = 303) of ATs reported never using PROMs in their clinical practice, 30% (n = 153) reported using them sometimes, 4% (n = 21) reported using them about half the time, 4% (n = 22) reported using them most of the time, and 2% (n = 8) reported using them always. Once these responses were dichotomized for comparisons, only 10% (n = 51 out of 507) of respondents reported using PROMs half the time or more (Table 3). There were no significant differences in PROM use based on highest level of education (P = .104), bachelor's degree type (P = .707), master's degree type (P = .818), year of professional degree completion (P = .825), or employment setting (P = .629). However, individuals who reported having access to professional and scientific journals beyond those associated with the NATA were significantly more likely to routinely use PROMs (13% [n = 39] versus 6% [n = 11], P = .016). The odds ratio for those with access compared to those without using PROMs was 2.3 (95% CI = 1.2–4.6).

Table 3 Percentage of Responders Who Reported Using Patient-Reported Outcomes Half the Time or More

View Fullscreen

Participants were also asked if their employer/enterprise had a policy concerning the collection of PROMs or other health outcomes. Those responding Yes (6%, n = 29) were significantly more likely to routinely use PROMS compared to those with no policy (81%, n = 409) or those who were unsure about a policy (14%, n = 69) (59% versus 7% and 7%, P < .001). Specifically, the odds ratio for those with a policy compared to those without was 13.6 (95% CI = 6.0–31.4).

Access to Resources for Evidence-Based Practice

Access to scientific journals beyond those associated with the NATA is summarized in the Figure. Access to journals differed significantly based on traditional employment setting, with a greater proportion of individuals employed in NCAA Division I settings having access to additional journals than did those employed at NAIA schools or secondary schools (P < .001). Participants also indicated up to 5 resources they used for learning about health care advancements and then ranked these recourses based on how commonly they used each indicated resource. These results are presented in Table 4. The most frequently indicated resource was NATA position statements, followed by local, state, or regional meetings and workshops; the NATA Clinical Symposia & AT Expo; and colleagues, friends, or coworkers.

View Figure

• Download as Powerpoint
• Download Figure

Table 4 Sources for Learning About New Advances in Health Care as they Relate to Athletic Training (Respondents Could Select Up to 5 Choices)

View Fullscreen

Impact of Education on Attitudes and Beliefs Toward PROMs

Advanced clinical training that is focused on athletic training was associated with increased awareness of and more positive attitudes and beliefs toward PROMs. This was evidenced by the higher Acquiring and Using and Total Attitude scores among individuals completing an athletic training–related postprofessional master's degree compared to those completing a non–athletic training, postprofessional degree. The impact of advanced, formalized clinical training was also supported by the higher Benefits scores observed among individuals who had completed a clinical doctorate. In 2013, CAATE released the “Standards for Accreditation of Post-Professional Athletic Training Programs,” which included the core competences of “Evidence-Based Practice,” “Quality Improvement,” and “Patient-Centered Care,” among others.¹¹ These core competencies emphasize the importance of linking current research, didactic instruction, clinical practice, and the unique personal and contextual factors of every patient. PROMs are central to many of these efforts, as they represent a method by which to evaluate outcomes, both within a single patient and across one's practice, or even compared to those described in other published literature.

Furthermore, PROMs create an opportunity to engage patients in the process and to solicit their individual feedback on their health. These core competencies have since been instilled in many athletic training master's and doctoral degree programs (both accredited and nonaccredited). The installation of the competencies may explain why those with a clinical doctorate demonstrated the highest Attitude and Belief scores and why those with an athletic training–related postprofessional master's degree demonstrated higher scores than those completing a non–athletic training–related master's degree. These results suggest that not all advanced degrees are created equal, and athletic training–related postprofessional degrees, including clinical doctorates, may better expose students to PROMs and their potential use. This observation is particularly important as the profession transitions to requiring a professional master's degree and as opportunities for formal postprofessional education and transition to practice may become limited.

There were no differences on any of the Attitude and Belief scales based on graduating before or after 2014, when the CAATE standards¹⁵ began requiring the inclusion of EBP and PROM-related knowledge and skills. Keeley et al¹³ and Hankemeier et al²⁸ similarly observed no effect for years of experience on beliefs or perceptions related to EBP. This lack of an effect for graduation year may reflect the multilayered educational strategy that was implemented to expand awareness of EBP and PROMs. Concurrent, formalized initiatives, aimed at expanding AT knowledge regarding EBP, were ongoing in 2013 to 2014 as new professional and postprofessional program standards went into effect and as EBP-specific continuing education requirements were initiated.^15–17 These broad and overlapping educational efforts may explain the overall positive attitudes and beliefs that we observed regardless of timing of professional degree completion.

Impact of Education on PROMs Use

Despite improving attitudes regarding PROMs, educational initiatives appear to have had little impact on the actual use of PROMs. As previously stated, the rate of PROM usage we observed does not represent an increase in adoption when compared to the results of previous literature. Furthermore, no differences in use were observed based on degree level, degree type, or completion year. This limited impact of educational interventions alone, particularly didactic interventions, is consistent with that noted in other health professions.^29–31 A systematic review²⁹ observed that traditional continuing medical education activities, such as conferences, were ineffective in altering physicians' professional practices, and more in-depth, practice-based interventions were necessary to meaningfully change behavior. Specifically within the rehabilitation sciences, educational interventions have been observed to have some short-term effects, particularly on knowledge and attitudes, but they fail to consistently result in the meaningful use of outcome measures in clinical practice.³¹

Within athletic training, previous research^13,32 has emphasized the importance of workplace support and clinical mentorship for the use of EBP. Our results suggest that until PROM education is occurring both in the classroom and in the clinic (both at entry level and postcertification), actual transfer of knowledge and awareness to practice may be limited. Experienced practitioners are necessary for impactful clinical mentorship³² to occur. Developing evidence-based practitioners should be a point of emphasis as both professional and postprofessional education continue to evolve. The implementation of immersive clinical experiences into professional education may present an opportunity through which practice-based educational experiences, under the guidance of strong clinical mentors, can help transition PROM knowledge to PROM use.³³ However, for this to be an effective path to increase PROM use, mentors who are actively using PROMs will need to be identified and/or fostered.

Barriers to PROMs Use

Previous studies³⁴ have established numerous barriers to the use of PROMs in allied health care. Although knowledge regarding PROMs has been reported as a barrier across professions,³⁴ it is not reported to be one of the top perceived barriers in athletic training.^9,11 More prominent barriers to the use of PROMs by ATs include the following reported perceptions: a lack of relevance to the AT's patient, negative time consumption for the clinician and/or patient, sense that information gained is not worth the effort, feeling that instruments are only useful for research, and sense that instruments do not improve clinical practice or treatment planning.^9–11 The combination of these perceptions of PROMs as a low-value undertaking and the lack of clear clinical standards for achieving patient engagement in a manner which documents the patient's perspective⁵ create strong barriers that must be addressed if we are to gain widespread use of PROMs in athletic training.

Attempts to adopt PROMs methodology from other clinical health professions may heighten barriers to widespread PROM use. For example, most existing PROMs that are common among physicians, physical therapists, and occupational therapists are primarily based on the performance of activities of daily living (ADLs). Athletes typically have less difficulty with ADLs, as demonstrated by the moderate to high ceiling effects often reported among ADL-focused questions on various PROMs.^35,36 Lam et al¹¹ reported that the most common PROMs being used by ATs were the Lower Extremity Functional Scale and the Disability of the Arm Shoulder and Hand (DASH). Both of these scales are heavily focused on ADLs, with minimal questions for high-functioning, physically active individuals who participate in sport. Therefore, these instruments may not provide unique, useful, and meaningful information to patients and clinicians in traditional athletic training settings. Although some instruments do include items addressing performance of higher level activities, these items are often nonspecific and may represent only a small sample of the total items contained within an instrument. For example, on the DASH only 2 of the 30 questions inquire about “recreational activities,” with listed examples including golf, hammering, tennis, frisbee, and badminton.³⁷ Second, differences in clinical models between athletic training and the fields of medicine, physical therapy, and occupational therapy should also be considered. Patients in non–athletic training fields are seen with less frequency and often during scheduled times. Less frequent visits allow time for changes to occur, which, in combination with requirements for reimbursement, likely contributes to the use of PROMs in these clinical settings being routine. The requirements for reimbursement create an environmental facilitator to the collection of PROMs. As evidenced in the present study, the greatest use of PROMs occurred among those with employers who had policies related to the collection of PROMs.

An additional environmental barrier that appears to restrict the use PROMs, and EBP in general, is a lack of access to scientific literature.¹³ Although use of PROMs was universally low, we observed that those with access to journals beyond those published by the NATA had double the odds of routinely using PROMs. While NATA publications are open access, many journals are not. It may be difficult to incorporate “the best available evidence” into clinical practice when access to it is restricted, as was reported by nearly 40% of respondents to our survey. It is particularly concerning that those working with some of the highest risk patients for long-term effects of injury, high school athletes, have significantly less access to journals beyond the NATA library than do those working in Division I college settings. NATA position statements (see https://www.nata.org/news-publications/pressroom/statements/position) were the most commonly indicated source for learning about new advances further demonstrating the importance of the coordinated provision of open-access information. While it is fortunate that these resources are used, we should also observe caution. Many position statements are years in the making and may go a decade or more without significant update. The continued growth of timely, open-access publishing and the building of partnerships between professional organizations to allow members reasonable access to high-level evidence from a variety of sources are paramount to ensuring disparities in care do not exist.

Future Directions

The results of the present study support that previous educational advancements have been effective in introducing ATs in general, and especially those with advanced clinical training, to the use of PROMs. However, there continues to be a gap between knowledge and practice. Strong clinical mentorship and structured transitions to practice may help bridge this gap. Furthermore, supporting preceptors by incorporating information related to PROM use and the availability of institutional scientific databases into preceptor training or working collaboratively to identify electronic medical records that incorporate PROMs may help bridge the knowledge-to-practice divide. The use of varied clinical placements,⁴ including placements in which the use of PROMs is a part of institutional policy, may also be beneficial. In particular, immersive clinical sites where PROMs are routinely used may provide in-depth, real-world opportunities for athletic training students to transfer knowledge to practice in a manner that persists into independent clinical practice.

Overall, our results, combined with existing literature, suggest that we cannot “educate our way out” of the lack of use of PROMs in athletic training clinical practice. Therefore, future research should look closer at developing strategies to address the barriers reported by practicing clinicians. First, efforts must continue to develop and validate outcome measures that are highly relevant to both clinicians and high-functioning, physically active populations. Second, policy and/or structural changes may be needed to aide and support the use of PROMs and the overall documentation of outcomes in athletic training clinical practice. Future interventions should look at more practice-based interventions, including policy changes and ongoing access to evidence-based resources. Other examples of potential structural or policy changes that might increase PROM use include, but are not limited to, the development of national databases, the incorporation of PROMs more broadly into common athletic training electronic medical record systems, or the linking of PROMs and other outcomes data to reimbursement or other incentives.

Limitations

As with any survey study, our results are limited by the interpretation and accuracy of our respondents. There is the potential that some respondents may have had access to additional journals beyond those associated with the NATA and simply not been aware of this access. Similarly, we did not ask the source of access to additional journals. For many, it likely related to employment based on institutional subscriptions, but it may also have been related to personal subscriptions, which could represent internal, individual differences and not environmental differences. We also did not differentiate between postprofessional accredited master's programs and other programs that respondents deemed “related to athletic training.” Given the voluntary nature of postprofessional CAATE accreditation (which is now being discontinued), we felt that such a division might be somewhat arbitrary. Instead, we left it up to respondents to indicate whether their master's degrees were related to athletic training or not. Likewise, engagement in advanced professional training may be a function of underlying personal factors that may also predispose an individual to engage in evidence-based practice and the use of PROMs. Finally, although we met our overall sample size target, we did not conduct an a priori power analysis. However, we conducted a purposeful sampling strategy intended to recruit ATs practicing in traditional secondary school and collegiate settings at all levels. The sample of participants with a clinical doctorate was small, representing just 2% (n = 10) of all respondents. However, given that in the 2018 NATA Salary Survey³⁸ only 0.8% of respondents working in secondary school or collegiate clinical settings reported having a clinical doctorate, we believe the sample is proportionally representative of those with a doctorate degree working in these settings. Nonetheless, these results should be interpreted with caution, given the small sample of this subgroup. It is likely a larger sample would yield additional significant findings, particularly related to PROM use, in this subgroup compared to others.