Content Validation of Athletic Therapy Clinical Presentations in Canada

COMPETENCY-BASED EDUCATION

Competency-based education in medicine and allied health care professions is common^1–3 and is a confluence of the knowledge, skills, abilities, and values of both teaching and assessment.¹ Competency-based education must integrate teaching and assessment methods that tie into the underlying philosophy, or they may end up working at cross-purposes. For example, a traditional lecture teaching method followed by a standard multiple-choice knowledge test may only require regurgitation of facts. On the other hand, teaching students how to work through clinical cases and then using scenario-based assessments does combine their knowledge, skills, abilities, and values, aligning with a competency-based educational paradigm. The key difference between the traditional and nontraditional teaching and assessment methods is that a competency-based paradigm uses a scenario or a clinical case as a foundation to both teach and measure a student's knowledge, skill, or ability. It should also be noted that teaching and assessment must be strategically aligned in curriculum planning.

METHODS

Since there was little existing evidence available regarding the use of CPs in athletic therapy education and a curricular change was desired, a consensus of experts was employed to validate⁶ the CPs chosen for this research. The modified Ebel procedure⁷ applied was originally designed for use in the measurement and evaluation discipline to content-validate high-stakes examinations where a minimal passing score is important. In the original procedure, test items were evaluated by experts for both content importance (0–100 with increasing importance) and ease of mastery (100–0 with increasing difficulty). Using these reversed scales, items deemed both easy to master (100) and important to the field (100) required a higher score for mastery (average of 100). On the other hand, items which were both difficult (0) and important (100), had a lower threshold for mastery (average of 50). Ebel himself declared the need for this type of content examination for competency-based tests when he stated, “[T]here is no escape from expert judgment in determining what ought to be measured and how it ought to be measured. The absence of this kind of test development is a frequent and serious weakness of many tests of professional or technical competence.”^7(p279) Therefore, the current study employed expert judgment to determine which of the proposed CPs should be included in a Canadian athletic therapy curriculum.

Expertise is a subjectively defined and operationalized construct, and this research sought to employ athletic therapy experts to verify the relative importance and difficulty of CPs. An expert, as described by Ericsson and Lehmann,⁸ can only reach that level of achievement once they have deliberately practiced their craft for 10 000 hours or approximately 10 years. Using their principles⁸ as guidelines, both local and national expert consultants were chosen for inclusion in this study. First, 5 local experts from MRU used existing literature to develop a list of 253 CPs thought to represent the scope of athletic therapy practice in Canada. Twelve national experts were also chosen to validate the CPs selected by the local experts. Six of these national experts were athletic therapists with at least 10 years' experience in teaching and clinical practice and were chosen to represent the remaining 6 CATA accredited institutions. Each of these subjects was further asked to recommend an additional clinical expert who had at least 10 years' experience as an undergraduate mentor working in clinical practice and also actively engaged with undergraduate students through both clinical teaching and student evaluations. All 17 experts met the inclusion principles outlined by Ericsson and Lehmann.⁸

Once the national experts agreed to participate in the study, they were e-mailed a link to an online survey hosted by Qualtrics (Provo, UT). Experts were given 30 days to complete the survey and were allowed to access their responses as many times as necessary to complete it. The survey presented the 253 CPs divided into specific body regions or emergency care conditions. Each expert answered the following 2 questions for each CP:

1. To what degree of IMPORTANCE do you think it is for an UNDERGRADUATE/BASIC level of COMPETENCE student to know the following CPs? Competence in this case would include the evaluation, management, and/or treatment/rehabilitation for: Example—Achilles tendinopathy.

2. How DIFFICULT would it be for an UNDERGRADUATE/BASIC level of COMPETENCE student to evaluate, manage, and/or treat/rehabilitate the following CPs for: Example—Achilles tendinopathy?

While the traditional Ebel procedure had experts rate importance and difficulty on a 3- or 4-point scale,^7,9 the current study modified the scaling method to use a 100-mm visual analog scale (VAS) instead. The VAS method was chosen because of its stronger statistical evidence to discriminate between expert judges^10–12 and because data can be reduced to quartiles during analysis. The CP importance scale was anchored by extremely important (100 mm) and irrelevant (0 mm; Figure 1). Difficulty, on the other hand, was anchored by extremely easy (100 mm) and extremely difficult (0 mm; Figure 2). Experts were asked to rate importance and difficulty for each CP (n = 253) by dragging an electronic cursor along the respective VAS. Furthermore, each expert's responses were blinded from the others, which was another modification from the traditional Ebel procedure. This blinding was borrowed from the Delphi method of consensus building to prevent dominance of 1 person's perspective over another.⁶

View Figure

• Download as Powerpoint
• Download Figure

View Figure

• Download as Powerpoint
• Download Figure

Data was tabulated inside Qualtrics and downloaded into Microsoft Excel (Redmond, WA) for analysis. An importance-difficulty index (IDI) was determined for each CP by calculating the mean of its importance and difficulty scales across all experts (ie, the mean importance score plus the mean difficulty score divided by 2). Since the IDI is a combination of importance and difficulty, it is a construct similar to a minimal passing score as outlined in the original Ebel procedure.⁷ However, this type of index has not been reported in the literature previously or employed in this fashion, to our knowledge, thus also making it another modification of the original Ebel procedure.

RESULTS

The mean importance and difficulty scores and the calculated IDI for each CP is provided in the Supplemental Material (available online at http://natajournals.org). Clinical presentation importance scores ranged from 99.3 for airway management (ie, most important) to 54 for high altitude cerebral edema (ie, least important). Clinical presentation difficulty scores ranged from 89.8 for biceps contusion (ie, easiest) to 21.2 for decompression illness (ie, hardest). The IDI for these CPs ranged from 93.4 for inversion ankle sprain to 37.9 for decompression illness.

Importance and difficulty in the current study were measured on a continuous scale. In traditional content-validation methods, importance and difficulty are often measured on a categorical scale (ie, extremely important, important, not important, and irrelevant).⁷ Items (or CPs in this case) that achieve 80% consensus among experts with the importance scale are kept, and those that do not achieve consensus are discussed.^13–15 In order to apply the traditional importance and difficulty scales to the current study, some translation was required.⁷ Therefore, continuous variables were translated into quartiles (25% segments) to coincide with a categorical scale. Specifically, CPs that fell in the first quartile (between 100–76) translated to extremely important on the importance scale and extremely easy on the difficulty scale. Values in the second quartile (between 75–51) translated to important and easy; values in the third quartile (between 50–26) translated to not important and hard; and values in the fourth quartile (between 25–0) translated to irrelevant and extremely difficult. Clinical presentations are presented by quartile for importance (Table 1), difficulty (Table 2), and the IDI (Table 3).

Table 1.  Importance Scale Results Broken Down by the Number of Clinical Presentations for Each Quadrant

View Fullscreen

Table 2.  Difficulty Scale Results Broken Down by the Number of Clinical Presentations for Each Quadrant

View Fullscreen

Table 3.  Importance-Difficulty Index Results Broken Down by the Number of Clinical Presentations for Each Quadrant

View Fullscreen

Closer analysis of the importance scale means revealed that no CP fell below 51 or important (Table 1). Therefore, if a traditional content-validation paradigm was employed, all CPs could be considered important enough to keep in the athletic therapy undergraduate curriculum. Closer analysis of the difficulty scale revealed that 1 CP (decompression illness) fell below 25 or extremely difficult and should be considered for removal from the curriculum (Table 2). The IDI is a new concept never before reported in the literature, as far as we know. The corollary of the IDI is the minimal passing score first introduced by the Ebel procedure. The IDI can be best be conceptualized by a 4 × 4 matrix with the 4 importance categories on the x axis and 4 difficulty categories on the y axis. All CPs must fall into 1 grid position, and those on the importance and easy grids should be kept, while others that are not important and/or difficult should be further evaluated for removal. Fifty CPs fell below the desirable IDI threshold of 51 (ie, the third quartile or lower) and may need more careful consideration when implementing them into the curriculum. Of those, 9 IDI scores (decompression illness, high altitude cerebral illness, high altitude pulmonary edema, acute mountain sickness, male penal avulsion, Pelligrini-Steida syndrome, ruptured tympanic membrane, hydrocele, and hematocele) fell into the not important and hard index in the IDI (Table 3). The remaining CPs in the IDI were all above the threshold of 51 and considered at least easy and important, thus concluding they should remain in the inventory of CPs (Table 3).

DISCUSSION

The current study employed 12 Canadian athletic therapy experts to judge the importance and difficulty of 253 CPs. The results indicate that all CPs are at least considered important. Thus, we could assume that all should be part of an undergraduate curriculum. However, closer analysis of both the difficulty scale and the IDI leave room for interpretation when considering these CPs in an undergraduate curriculum. For example, when this list is cross-referenced with the 9 CPs that fell below the important and hard threshold, some of these may be considered optional for some undergraduate programs. The 9 CPs that fell into this category appear to have geographical or environmental relevance. Regardless of the scale or index chosen, there are a number of CPs that create questions and discussion regarding their implementation into an undergraduate curriculum.

There may be other challenges associated with implementing a CP model of curriculum delivery as well. For example, is there an optimal timing or sequencing for introduction and development of competence of these CPs? It does not seem realistic to introduce students to the entire list of CPs in their second year due to sheer volume. Are some CPs better suited to earlier introduction in the program and others better suited to later in the program? Another question to be addressed in future research is whether the IDI rating for each CP can help guide the timing when these can be introduced.

Papa and Harasym¹⁶ identified 5 distinct curricular delivery models that have led curriculum reform in medical education: (1) 1765—the apprenticeship model; (2) 1871—the discipline-based model; (3) 1951—the organ-based model; (4) 1971—the problem-based model; (5) 1991—the CP-based model. The last 2 models are grounded in a constructivist approach to learning and of particular interest in this study. Schmidt et al¹⁷ identified the link between problem-based learning (PBL) and the constructivist approach that underpins it. The problem or case represents a part of the world that must be understood by a student, and by working through the problem, they construct a theory of diagnosis, management, and treatment of the problem.¹⁷ A PBL or case-based learning model is thought to have a common set of principles¹⁸: (1) the study of clinical cases; (2) small group discussion; (3) collaborative independent study; (4) hypothetical-deductive reasoning; and (5) a style of faculty direction that concentrates on group process rather than imparting information.

The CP model of curriculum delivery has a subtle but different approach to teaching clinical cases compared to traditional PBL methods.^16,19 The first difference is that rather than using deductive reasoning, it uses inductive reasoning, whereby the student starts with a clinical case or diagnosis. From this point, students are presented with a schema that outlines the steps an expert would have taken to come to the diagnostic, management, and/or treatment conclusion. Metaphorically, it is a roadmap created by an expert through charted territory. A PBL approach, on the other hand, may give someone instructions on how to get from 1 location to another by listing the turns to make, but not show them the map for full perspective. A second difference is that PBL claims that the problem-solving approach employed with 1 case is transferable to other cases.⁶ However, this assumption has been refuted by some.^20,21 Regardless of the controversy in the transference from 1 CP to another, the notion of using cases to guide learning is central to developing competency in undergraduate athletic therapy.

Developing the various expert schemata that are used to guide novices through their learning experiences presents another challenge when implementing the CP model into the curriculum. Clear terminal objectives must be established to guide learning. Furthermore, being competent should have contextual qualifiers rather than presenting a static construct with a binary response: competent or incompetent.¹ For example, the development of manual therapy skills to manage soft tissue manipulation or joint mobilization may be 1 of those skills that fit into the qualified competence construct. Students may have a qualified competence whereby they can recognize when to apply joint mobilization, how to place their hands, the theory behind performing joint mobilizations, and even be able to perform the joint mobilization in a crude, rudimentary fashion. However, they may not have honed the psychomotor skill required to feel the precision necessary to be 100% effective the day they graduate. Therefore, it will be important for graduates to realize that their learning does not end once they have completed their program, but that with more practice and time, they can refine their skill and move further along the novice-expert continuum.

The ultimate goal of any medical or allied health care professional program is move students from novice to expert in the most efficient and effective manner possible. Dreyfus²² developed a model and continuum that describes a stepwise progression from novice to advanced beginner to competent to proficient to expert. We proposed a modified version of the Dreyfus model (Table 4) specific to the athletic therapy discipline²³ for discussion and future research considerations. In order to accomplish competence by the time an athletic therapist graduates from their undergraduate program, careful planning and constructive alignment between curriculum philosophy, delivery, and assessment is necessary.

Table 4.  The Dreyfus Novice-to-Experts Model of Development for Athletic Therapy (Adapted from Balataden et al, 2002, p. 106)

Abbreviation: CP, clinical presentation.

View Fullscreen

The 3 competence stages that are of most interest to undergraduate educators are novice, advanced beginner, and competent. There are some who would argue it is challenging to progress students from novice to competent in the time permitted by most undergraduate or professional programs.^24,25 Moreover, how can programs ensure athletic therapy program students are competent upon graduation? There are many questions and challenges when implementing competency-based curricula, and those associated with CPs are no different.

There are a number of limitations of this study that frame its interpretation. A method for content validation that is typically employed for student evaluation was employed to content-validate CPs. Therefore, rather than specific items that are content valid, the claim in this study is that these CPs are content valid for the Canadian athletic therapy context. A method to establish CPs for the Medical Council of Canada (MCC) was employed whereby a group of experts on a task force created the list.²⁶ We feel the approach taken is equally as effective as the method undertaken by the MCC. Another limitation related to the method employed was the choice of experts across Canada. It may be a biased sample, and empirical results listed in the Supplemental Material may vary with a different set of experts. However, the potential pool of experts is limited, particularly as it is related to Canadian athletic therapy educators. Furthermore, the list of CPs in this study should be considered a starting point. Over time, it is conceivable that CPs will be added or removed from this list, just as the MCC list of CPs has evolved over time. Finally, the IDI has never been employed in this manner previously, as far as we know. Interpretation of this index is open for debate. It is possible that the number calculated for each CP represents the construct similar to the minimal passing score. However, since there are no written or practical examination items associated with these numbers, it would be important to revisit these calculations once those evaluation tools have been content-validated.

CONCLUSIONS

In closing, 253 CPs are presented and thought to be representative of the athletic therapy scope of practice in Canada. Clinical presentations have not been developed in the athletic therapy context prior to this research, to our knowledge. Therefore, we anticipate there will be more debate on the content of this list. We also anticipate a great deal will be learned once these CPs are employed as a vehicle to help align teaching, learning, and assessment within the competency-based framework expected for athletic therapy programs in Canada.