Content Validation of the Athletic Training Milestones: A Report from the Association for Athletic Training Education Research Network

Design

We used a formal CVI, as described by previous researchers,^2–5 to determine the content validity of the Athletic Training Milestones. Content validity is a critical component for scale development of high-quality instruments to ensure a sufficient number of items to appropriately assess the content of interest.⁷ Our intent in conducting a CVI of the Athletic Training Milestones were to assess whether this evaluative tool is a valuable measure to appropriately measure ATs' progression of independent knowledge, skill, and behavior acquisition from novice to proficient with aspirational goals of becoming an expert.³ This study was deemed exempt research by the A.T. Still University Institutional Review Board.

Participants

As highlighted by Grant and Davis,⁴ the validation process is heavily influenced by how the content experts are chosen and used. Content experts are often selected on the basis of their training, experience, and qualifications. More specifically, content experts may be selected due to their own research experience on the topic,³ clinical expertise,⁴ or expertise related to the conceptual framework of the instrument.^2,4 Whereas some debate exists about the number of experts needed, earlier investigators have suggested between 3 and 10 content experts are necessary to conduct a CVI^2,3; including more than 10 content experts is considered highly conservative. Because it can be difficult to find content experts who meet all criteria—that is, research experience, clinical expertise, and theoretical expertise of the content included in the instrument—a range of representation across the expertise criteria should be sought among those asked to serve as content experts.⁹

To ensure we had enough content experts to conduct a highly conservative CVI and account for participant attrition for this study, we recruited a purposeful criterion sample of 31 health care professionals who primarily served as clinicians and educators and were known among our professional network for their experience in implementing the Athletic Training Milestones for personal-, student-, or peer-evaluation use. Of the 31 individuals recruited, 24 were willing to participate, 1 indicated unwillingness to participate, and 6 did not respond to the initial request.

Instrumentation

To conduct the CVI, a rating score document was developed using Excel (version 16.61; Microsoft Corp). Each general competency (patient care and procedural skills, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, systems-based practice) was set up as an individual tab in the document. Within each tab, the individual subcompetency items (eg, demonstrates humanism and cultural competency, responds to each patient's unique characteristics, needs, and goals) were listed, followed by a field to input the rating score and a field to insert comments if warranted. For scoring, we adopted the 4-point item-rating continuum advocated by Davis⁹: 1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, 4 = highly relevant (Table 1).

Table 1 Specific Scoring Language Used for the Athletic Training Milestones Content Validity Index

View Fullscreen

Procedures

In February 2021, we sent a recruitment email to the criterion sample. The email contained a brief introduction and purpose of the study, a general overview of the CVI process, the estimated time to complete the CVI (approximately 2 hours), and a request to participate. Once individuals responded and indicated a willingness to participate, they were sent an email with 9 file attachments: CVI process instructions, CVI scoring template, the introduction and preface of the Athletic Training Milestones document (https://www.atmilestones.com), and the subcompetency milestones of each of the 6 general competencies separated into individual files. The email also included a URL link to a brief, web-based demographic questionnaire aimed at capturing general demographic information about each content expert who completed the CVI. Each person was given 4 weeks to complete the CVI, and we sent 2 reminder emails during the data collection period.

To begin the CVI process, the content experts were asked to familiarize themselves with the 9 documents and to review the introduction and preface of the Athletic Training Milestones. The intent of this review was as a refresher to ensure that all content experts were extremely familiar with how the Athletic Training Milestones document was formatted, the terminology used, and the descriptions of the different performance levels included. Once familiarized, content experts were asked to rate each general competency. For each subcompetency item within the general competency, individuals were instructed to read every criterion listed in each column, starting with the criteria in the critical deficiencies column, and then progress to the right through the criteria in the level 5 column. Next, they were instructed to identify a rating score on the basis of whether the criteria listed for progression from 1 level to the next were relevant or whether critical knowledge or skills were missing from the criteria. Individuals were told to base their score only on the overall progression from the lowest level to the highest. Furthermore, if a score was <4, they were asked to use the comment box next to that score to provide more information regarding the score selection. This process continued until all 28 subcompetencies within the 6 general competency areas were scored. The final rating score document was emailed to the principal investigator (C.W.B.).

Data Analysis

We received completed rating score documents from 21 of the 24 individuals who agreed to participate. Before data analysis, all individual rating scores were deidentified and reviewed for appropriateness. For inclusion in the final data analysis, we used the following criteria: (1) a self-perceived rating of moderately or extremely familiar with the Athletic Training Milestones, (2) submitted rating scores for all 28 subcompetency items, and (3) followed instructions to base scores only on the overall progression from the lowest level to the highest. Upon review, we found that the rating scores from 9 people did not meet all criteria and were excluded; 1 respondent self-reported being minimally familiar with the Athletic Training Milestones, 2 respondents did not submit rating scores for all 28 subcompetency items, and 6 respondents did not follow the instructions for scoring. Common reasons given for not following the instructions included lowering of the score for grammatical considerations or items unrelated to clinical relevance. All final rating scores were merged into a single Microsoft Excel document for data analysis.

We conducted CVI analyses at the individual item, section, and overall scale levels. To compute the individual CVI (I-CVI) for each subcompetency item, we tallied the number of experts who gave that item a rating of 3 or 4 and then divided by the total number of experts (n = 12).^2,6 When ≤5 experts are involved, the I-CVI must be 1.00^2,6; with >5 experts, an acceptable I-CVI must be above 0.78.³ To calculate the scores for the 6 general competency sections, we tallied the individual I-CVI scores and then computed the average based on the number of items in the respective general competency section.

The CVI of the overall scale (S-CVI) is often determined using 1 of 2 methods. The most straightforward approach is to calculate the average (S-CVI-Ave) from all individual-item I-CVI scores. In the past, authors^3,4,9 have indicated that an S-CVI-Ave of ≥0.80 is acceptable; however, Waltz et al⁵ noted that the standard criterion for the S-CVI-Ave, also called the average congruence percentage, should be 0.90 to help account for chance agreement. Another, more conservative approach to calculating the S-CVI is to require universal agreement (S-CVI-UA) among experts. The S-CVI-UA therefore focuses on the proportion of items within an instrument that achieved an acceptable rating (ie, 3 or 4) by all content experts.² This approach is considered overly stringent and is less likely to be used with >2 content experts given that universal agreement by all content experts becomes more difficult as their number increases.³ Furthermore, because it is a much more conservative approach for determining scale validity, experts indicate a minimum of 0.80 is acceptable for S-CVI-UA.⁵

To ensure transparency of all possible calculations for the S-CVI of the Athletic Training Milestones, we calculated both the S-CVI-Ave and S-CVI-UA. We computed the S-CVI-Ave by tallying the I-CVI scores for all 28 subcompetency items and then dividing by the total number of individual items. We used the 0.90 threshold identified by Waltz et al⁵ as our minimal acceptable limit for the S-CVI-Ave. We determined the S-CVI-UA by tallying the number of individual items that had an I-CVI score of 1 and then divided by the total number of individual items.