Accuracy of Undergraduate Athletic Training Student Performance Factors for Determining Board of Certification Exam First-Time Pass Outcomes

Experimental Design and Participants

A retrospective casual comparison (ex–post facto) quasi-experimental design was conducted at a R1-classified university. The university's Office for Research Protections determined that this study did not meet the definition of human subject research and therefore did not require further review or approval by the institutional review board.

A power calculation was performed to determine the sample size needed to assess the accuracy of athletic training student factors as indicators of first-time success on the BOC exam. Per receiver operating characteristic (ROC) curve analysis, using the following parameters—single test method; significance level (α) of 0.05; power (1−β) of 80%; area under the ROC curve (AUC) of 0.80; and allocation ratio of 0.23—a respective minimum sample of 25 and 6 occurrences for passing and failing the exam were required to execute this study.¹⁰

Data Collection

An administrative support assistant provided the investigators with deidentified secondary program admissions data and corresponding BOC exam results from a sample of 43 records for students who had recently graduated from the undergraduate athletic training major. Considering the retrospective nature of our study, the records were consecutively included for analysis to prevent selection bias. Records that were missing data points for the variables of interest were excluded from the study. A brief accounting of the sample population's demographics and related first-time BOC exam results are described in Table 1.

Table 1 Cohort Demographics and Associated First-Time Board of Certification (BOC) Exam Outcomes (n = 43)

View Fullscreen

Secondary admissions data consisted of cumulative GPA upon entry to the major, preprofessional-phase course GPA, averaged admission interview scores, averaged preceptor evaluation of applicant scores, and a unique composite score inclusive of the aforementioned variables. The content areas of the 4 courses calculated in the preprofessional-phase course GPA included introduction to athletic training, functional human anatomy, foundations of clinical practice, and acute care and emergency response. Questions included in the admission interview primarily addressed the applicants' strengths, communication skills, interest in the athletic training major, and where they believe the profession fits under the health care umbrella (Figures 1 and 2). Furthermore, the categories of the preceptor evaluations included professional appearance, communication skills, sports health care administration, preventative care, basic clinical skills, injury recognition, work ethic, and personal attributes (Figure 3). The composite score was configured of weighted factors and was developed to provide the program's admissions panel with a simple, single ensemble metric of applicant viability for entrance to the major. Various athletic training and other allied health care programs incorporate such a metric in their admissions process.⁶

View Figure

• Download as Powerpoint
• Download Figure

View Figure

• Download as Powerpoint
• Download Figure

View Figure

• Download as Powerpoint
• Download Figure

View Figure

• Download as Powerpoint
• Download Figure

Grade point averages were calculated on a 4-point scale, interviews were scored out of 10 points, averaged preceptor evaluation of applicant scores were out of 100%, BOC exam outcomes were tallied on a pass or fail basis, and the unique composite score ranged from 0 to 10. Descriptive statistics for the secondary program admissions data are provided in Table 2, and Table 3 outlines the composite score format.

Table 2 Descriptive Statistics for Secondary Program Admissions Variables (n = 43)

View Fullscreen

Table 3 Point Values and Associated Weights Assigned to Components of the Unique Composite Score

View Fullscreen

Statistical Analyses

We utilized IBM® SPSS Statistics 25 (IBM Corporation, Armonk, NY) software to perform ROC curve analysis for each variable of interest. The accuracy of each factor was determined by the AUC and interpreted according to Carter et al,¹¹ as detailed in Table 4. Using the ROC curve data, optimal thresholds for passing the BOC exam on a first attempt were computed for each variable by the index of union (IU) method. The IU method identifies an optimal threshold from the ROC curve that possesses values for both sensitivity and specificity that are close to that of the AUC while minimizing the difference between sensitivity and specificity.¹² Additionally, positive- and negative-likelihood ratios were computed from the sensitivity and specificity of each optimal threshold to aid in translating our outcomes. We interpreted likelihood ratios according to the method of Jaeschke et al,¹³ as described in Table 5. Data were evaluated for normality, randomness, and variance to ensure statistical conclusion validity.

Table 4 Values for Interpreting the area under the curve (AUC), from Carter et al¹⁰

View Fullscreen

Table 5 Practical Values of Likelihood Ratios Adapted from Jaeschke et al¹³

View Fullscreen

Secondary Admissions Criteria

Preprofessional-phase course GPA was a more accurate secondary admissions criterion for gauging first-time success on the BOC exam than was the hypothesized cumulative GPA upon entry to the major, which may be the product of our pilot data being potentially underpowered. This finding may also reflect the considerable changes made to the BOC exam and educational competencies since a prior investigation was conducted.⁵ Furthermore, our results complement those of Ennulat et al,¹⁴ who found that a specified prerequisite course GPA was a greater estimator of success on the Physician Assistant National Certifying Exam (PANCE) than was cumulative GPA. Exposure to professional competencies, and thus credentialing exam content, provided in the preprofessional (or prerequisite) courses, and absent from general education courses, may lend to this finding. Additionally, students' interest in the profession and health care–related content of the preprofessional phase, in comparison with the differing content areas of required courses included in the cumulative GPA, may underpin our results.

Contrastingly, the averaged admissions interview score and averaged preceptor evaluation of applicant score were found to be inaccurate and poor indicators of first-time success on the BOC exam, respectively. Similarly, Higgins et al¹⁵ found interview scores to be a predictor of PANCE success for only 2 of the 6 programs included in their study. Therefore, program personnel may consider prioritizing student performance in prerequisite courses followed by general education courses with regard to athletic training admission decisions to help ensure first-time success on the BOC exam and compliance with Standard 11. Subsequently, our findings suggest that scored interviews and appraisals require thoughtful consideration for their utility in the admissions process, apart from BOC exam outcomes, given the time burden associated with capturing these variables. For instance, interviews and appraisals may be of other benefit, such as in evaluating intangible skills, though programs must delineate their intended function. This is especially applicable given that communication skills, trustworthiness, dependability, and commitment to the field have been identified as characteristic markers of quality athletic trainers.^16,17 Therefore, these variables may better reflect a student's penchant for the profession, as opposed to first-time BOC exam success.

Unique Composite Score

Prior evidence⁶ suggests that a composite set of variables including overall academic GPA, athletic training–specific GPA, academic minor GPA, ACT composite score, and number of semesters predict first-time exam success. However, this was established for a previous iteration of the credentialing exam and only accounted for 42% of the variance in predicting first-time BOC exam success. We found our unique composite score to be 80% accurate. Although our ensemble metric was 38% more accurate than that of Harrelson et al,⁶ it was on the low end of “good” accuracy. This may be due to the limited accuracy of the interview and preceptor evaluation in determining this outcome. Comparatively, the difference among the preprofessional-phase course GPA, which was the most accurate criterion, and the unique composite score was within 1%. Despite being poor indicators of successfully passing the BOC exam on a first attempt, these factors appear to assess intangibles significant to athletic training professionals, suggesting that the unique composite score can be utilized to comprehensively gauge the professional and academic preparedness of students. The findings of this study suggest that athletic training specific course GPA should be the emphasis of an ensemble metric, as the accuracy of the unique composite score was attributed to this GPA outcome. The results of prior pilot analysis determined the weights we applied to each component of the unique composite score; however, weighting of components should be continually reexamined and reconsidered or rescaled as appropriate. Future research should be conducted to determine weighting of such variables and the potential inclusion of other applicable indices.

Post Hoc Analysis of Additional Academic Variables

We conducted a post hoc analysis for academic variables that have been previously investigated as indicators of first-time BOC exam success. This analysis was conducted in order to account for the current version of the BOC exam and to provide additional variables for programs to consider when devising admission processes. These additional factors included cumulative GPA upon graduation, combined math and reading SAT scores, and clinical education hours. From the original sample, 33 records contained the inclusive data for our post hoc analysis. Descriptive statistics for these additional academic variables are provided in Table 7.

Table 7 Descriptive Statistics for Additional Academic Variables (n = 33)

View Fullscreen

The combined math and reading SAT score provided excellent accuracy. This variable had a higher sensitivity, suggesting that it is better suited to identify students who may fail the BOC exam on their first attempt; furthermore, related likelihood ratios suggest a combined math and reading SAT score of 960 yields a small shift in the probability of passing and a moderate shift in failing. Cumulative GPA upon graduation and clinical education hours provided poor accuracy. Cumulative GPA upon graduation had a higher sensitivity, suggesting that it is better suited to identify students who may fail the BOC exam on their first attempt; moreover, associated likelihood ratios suggest a cumulative GPA of 3.19 upon graduation yields a very small shift in the probability of either passing or failing. Meanwhile, clinical education hours had a higher specificity, suggesting it is better suited to identify students who may pass the BOC exam on their first attempt; furthermore, related likelihood ratios suggest a total of approximately 1131 hours yields a very small shift in the probability of either passing or failing. The ensemble outcomes of the ROC curve analysis for the additional academic variables are found in Table 8.

Table 8 Ensemble Outcomes of Receiving Operating Characteristic (ROC) Curve Analysis for Additional Academic Variables (n = 33)

View Fullscreen

Our findings suggest that SAT outcomes may provide an early indication of student acuity, as the combined math and reading SAT score delivered the greatest accuracy of all examined variables. Although they vary in content, both the SAT and BOC are standardized exams, potentially lending to this excellent accuracy. Prior investigations in the sister disciplines of physician assistant (PA) and physical therapy (PT) have utilized the standardized Graduate Record Examination (GRE) to predict success on their respective credentialing exams. The outcomes of the GRE have proven^18,19 to be a reliable predictor of success on both the PANCE¹⁵ and National Physical Therapy Exam (NPTE). Therefore, athletic training programs at the undergraduate level may consider incorporating SAT outcomes in their admission decisions, while programs that already have or intend to transition to the entry-level master's may utilize GRE outcomes. Furthermore, the inclusion of either SAT or GRE outcomes in a composite score may increase the accuracy and utility of such a variable in program admission decisions. Future research should be conducted to determine the appropriate weight such outcomes should be allocated within a composite score.

Although cumulative GPA upon graduation provided poor accuracy, it has been cited²⁰ as a common measure utilized to gauge student quality. Prior investigations have revealed cumulative undergraduate GPA as contributing to and indicating success on the PANCE^14,15 and NPTE^18,19,21 as well as the BOC exam,²² reflecting its utility in graduate health care programs. Utzman et al¹⁹ identified an undergraduate GPA of 3.49 as an optimal threshold indicative of NPTE success. Furthermore, Bruce et al²² identified an undergraduate GPA of 3.18 as an optimal threshold indicating success in an athletic training master's program. Although their study did not identify an undergraduate GPA threshold specific to first-time BOC exam success, the 3.19 threshold identified in our study suggests that graduate-level athletic training programs may consider elevating the minimum GPA to be above the common industry standard of a 3.0.^20,23

A concluding post hoc analysis consisted of examining the influence of all variables as predictors of first-time BOC exam success using a stepwise regression approach. The results of this assessment produced the following regression equation: −173.1 + 85.5 preprofessional-phase course GPA + 0.029 clinical education hours + 0.199 combined math and reading SAT scores. The corresponding coefficient of determination (R²) was 76.69%, with preprofessional-phase course GPA accounting for 38.23% of the variance, followed by 37.25% for combined math and reading SAT scores and 1.20% for clinical education hours. Our post hoc regression analysis complements our initial ROC curve outcomes in that preprofessional-phase course GPA and combined math and reading SAT scores were the overall strongest predictors of first-time BOC exam success. Consequently, there was a high degree of collinearity between preprofessional course GPA and the unique composite score (r = 0.842), reinforcing the practical utility of the latter original admissions metric as an all-inclusive or holistic indicator of first-time BOC exam success.

While the results of our study may be employed as evidence with which to consider changing or bolstering admission requirements for athletic training professional programs, the realization of such efforts will likely require review by, and discussion with, administrators at various levels of an institution's hierarchy. It is reasonable to assume that admission requirements are often analyzed in the context of overall student enrollment and the specific mission or philosophical framework of a respective institution.²⁴ Currently, trends suggest that enrollments in American higher education institutions are declining, as noted by a decrease of 1.8% in 2018 and of 1.7% in 2019, with community colleges and 4-year public universities seeing the greatest drop,²⁵ although enrollment in postbaccalaureate degree programs are on the rise.²⁶ Hence, personnel should be cognizant of such variables that may influence their particular admissions model and strategically plan or adapt to help manage challenges inherent to their individual settings as a means to facilitate student and, thus, program success.

This study was descriptive in nature, and the sample was exclusive to our institution; therefore, program personnel are cautioned that applying the exact same strategies and tactics to their framework may not be feasible. In light of this, we are confident that our original work provides educators and administrators with basic concepts and advanced approaches to consider in modeling or revising their curriculum to promote overall program efficiency and quality. These mainly consist of encouraging a balanced, evidence-based, or informed, as opposed to anecdotal, perspective to drive policies and practices. Use of such a strategy will help mitigate forms of bias in what is a very complex process of operating academic programs that are subject to disparate individual administrative, faculty, and staff viewpoints. This may be especially applicable in instances in which contrasts arise in the utilization of quantitative and qualitative metrics to determine admissions.²⁷ Correspondingly, we provide educators with explicit tools with which to weigh quantitative and qualitative data from their cohorts to establish sound minimum thresholds for such variables and to devise an all-inclusive criterion, such as our unique composite score, that intends to reflect the spirit of a holistic process.

We trust our data may also be used to realize additional efforts through which to better prepare students for sitting for the BOC exam. Given the strong association between the SAT and BOC exams, which are standardized high-stakes assessments, educators may elect to identify students who did not perform well on the SAT and explore the elements that underpin this performance, such as test anxiety, and implement strategies to offset such negative influence. Additionally, performance in preprofessional-phase courses may be an advantageous benchmark with which to recognize students who may experience difficulties passing the BOC exam on a first attempt. Thus, remediation plans for those students at risk may be contemplated as a means to improve the likelihood of BOC exam success, as has been suggested²⁸ in other allied health care professional programs. Moreover, personnel tasked with delivering the BOC exam may find the outcomes of our study helpful in assisting in their charge to continually shape the evaluation to be a valid reflection of candidate clinical competency. The intertwining of educator and assessor standpoints in this undertaking may contribute to elucidating the intrapersonal and interpersonal factors that advance competency learning and appropriately evaluating such skills, which are requisite for success in a profession.²⁹

Findings of this study require careful consideration with application to the current transition in entry-level degree to a master's. SAT score and the specific GPA measures analyzed in this study may not be broadly represented among entry-level master's programs' admissions criteria. However, our findings suggest that performance on standardized exams required for entry to institutions, such as the GRE, may be considered a potential influential factor in forecasting student first-time pass success on the BOC exam, which has implications for program compliance with accreditation standards. Accordingly, this particular phenomenon has been observed in related studies specific to PT education,^18,19,28,30 in addition to the predictive power of prerequisite GPA^18,19,21,30 and post-admission GPA.²⁸ Thus, the findings of this study ultimately lend to developing an evidence-based practice basis for admissions criteria that programs may consider with on-boarding an entry-level master's degree in athletic training.

Limitations

Some limitations exist for this study. Primarily, the data were collected from a sample of students from a single R1-classified university and retrospectively analyzed. This institutional bias may prevent our findings from being generalized to programs nationwide. Variances in athletic training program and degree requirements across universities may also limit generalization of these study findings.

The admission interview panel and preceptors providing evaluations in our program were inconsistent; therefore, the levels of expertise and years of experience in evaluating students may have varied. While this limits the internal validity of our study, it may in fact bolster external validity. Additionally, the clinical experiences of the preprofessional phase are observational in nature, meaning that preceptors can only speculate upon the success of the applicant as an athletic training student. Although preceptors can gauge the students' interest and foundational knowledge, these evaluations do not reflect the application of knowledge, skills, and abilities that are acquired and demonstrated further along in the athletic training curriculum. Hence, program administrators and faculty should view the outcomes of these assessments as supplementing the academic success of a student before admission decisions are made.

An additional factor potentially influencing our results is the consistency of course instructor. Two of the 4 preprofessional courses are offered solely at the university's main campus and are routinely taught by the same instructors during the semester that immediately precedes admission to the athletic training program, which provides consistency in course content and delivery for each incoming cohort. Contrastingly, general education courses can drastically vary with regard to campus offering, course instructor, and the semester in which the student takes such courses. Therefore, course consistency and content may be factors lending to the greater accuracy we observed for the preprofessional-phase course GPA.

Lastly, the outcomes of this study are primarily focused on bachelor degree programs; thus, the comparisons drawn to PA and PT programs require careful consideration. However, the results of this study may lend insights into athletic training programs as they continue to transition to the entry-level master's degree.