Integrating Clinical Prediction Rules Throughout the Curriculum

Derivation of CPRs

When teaching CPRs it is important for students to understand how they are developed because that will aid in the critical appraisal process and will help them decide whether the CPR is appropriate to implement with patients. Creating a CPR begins with selecting and defining both the predictor and the outcome variables.^1,5,12,19 Predictor variables are those factors believed to be helpful in identifying the suspected condition. For the majority of CPRs, these predictors come from examination findings (eg, positive Lachman test or point tenderness) or patient history (eg, history of previous injury or age) and may be included initially on the basis of either previous research or clinical experience.^5,19 The process of narrowing the list of predictors is typically done with a statistical analysis of the relationships between the predictor variables and the outcome variable using correlation and regression statistics. Logistic regression models are used to identify predictor variables that can be removed without reducing the predictive nature, and then a receiver operating characteristic analysis can be used to look at the prediction accuracy of the group of predictor variables.¹⁸ Predictor variables that have stronger relationships with the outcome variable are included and then grouped together. Narrowing the list to a few predictor variables is important because it keeps the CPR easy to use and understand and decreases the number of participants who must be evaluated in the original derivation study. However, there are some instances in which capturing all potential predictors is imperative (eg, situations in which an incorrect diagnosis could prove fatal).⁵ Childs and Cleland⁵ suggest that in these instances, researchers and clinicians should collect data on as many predictor variables as necessary to ensure they do not exclude any variables that may prove helpful.

When developing a CPR, often the outcome variable is the presence of an injury or condition or a desired treatment outcome. Outcome variables should be well defined and specific, meaning an outcome variable such as “knee pain” would not typically be appropriate.^3,18,20 If the outcome variable lacks clarity or is not easily applied by multiple clinicians, the CPR might not be used appropriately. After outcome and predictor variables are chosen, the CPR should be applied prospectively to patients or, if a prospective approach cannot be taken, applied to patient cases retrospectively. If patient records do not include information relevant to all the predictor and outcome variables, they should not be used. Prospectively, patients should be evaluated or treated using the CPR and the current criterion standard simultaneously. The criterion standard serves as a reference and determines whether the patient has the condition of interest.⁵ For the Ottawa Ankle Rules, the criterion standard is radiography (ie, x-ray).⁷ In order to evaluate the accuracy of a CPR, it is recommended that the CPR be implemented with approximately 10 to 15 participants for each predictor variable.^3,5,19,21 Failure to include an appropriate number of participants could result in the study being underpowered, which can lead to missed diagnoses or inappropriate treatment.²² In order to minimize bias, the clinician who is responsible for collecting data regarding the predictor variables should not be the clinician applying the criterion standard, and the individual responsible for applying the criterion standard should be blinded to the results of the initial examination.^3,5

Once the data from all participants have been collected on the predictor variables and the criterion standard reference, the results must be analyzed. This analysis is typically done using backward logistic regression statistics, which are designed to evaluate the ability of a set of predictor variables to predict an outcome.^5,18 Including appropriate variables and completing statistical analysis allow clinicians to create CPRs but do not allow them to judge the CPR's accuracy. Once the list of predictor variables has been narrowed, the CPR is assessed for accuracy by comparing the results of the CPR with that of the criterion standard. Like most diagnostic tests, the accuracy of a CPR can be evaluated using sensitivity and specificity. Understanding the sensitivity and specificity of a CPR is helpful, but one cannot be sure how useful CPRs are without completing a validation study.

Validation of CPRs

After derivation, a CPR should be validated to ensure the results were not simply the result of chance. Validation studies should be used to evaluate the CPR in a different setting with different participants and/or clinicians. The Table shows the four levels of validation typical to CPRs. When a CPR is first created, it is at the lowest level of validation (Level IV). The validation level increases as the CPR is studied across a variety of patients and clinical settings. As the level of validation increases, clinicians can use the CPR with more confidence that implementing the CPR will have an impact in their clinical practice.^12,13,22 Recently, David et al²³ evaluated the predictive value of the Ottawa Ankle Rules in an athletic population and discovered a high negative predictive value (ie, negative results can rule out a fracture) but a low positive predictive value (ie, positive results cannot rule in a fracture). The researchers'²³ attempt to evaluate the validity of the Ottawa Ankle Rules demonstrates the importance of validating CPRs in specific settings with specific patient populations, because their results were somewhat inconsistent with previous Ottawa Ankle Rule literature.^6,7,10 The nature of the way CPRs are created and validated makes each rule specific to a certain group of people and variables, and their results inform health care decisions.

Table Levels of Clinical Prediction Rule Validation

View Fullscreen

Impact Analysis of CPRs

Validation indicates that a CPR is reliable and accurate, but it does not tell a clinician whether putting the CPR into use will truly have an impact on clinical practice. For that reason, the final step in developing a CPR is completing an impact analysis.^5,12,13 The impact analysis allows researchers and clinicians to gain information regarding how the CPR influences practice patterns or the outcomes and costs associated with patient care.⁵ Ideally, an impact analysis is conducted in the form of a randomized control trial which allows for one group of patients to receive the standard of care while another group receives care based on the CPR. Unfortunately, a randomized control trial is not always appropriate, and therefore, impact analyses can also be completed by assigning different sites of care to different treatment groups (eg, athletic training facility A uses the standard of care and athletic training facility B uses the CPR) or by comparing similar outcomes from the same site before and after CPR implementation.⁵ An impact analysis assesses the ways in which implementing a CPR influenced factors such as patient outcomes, satisfaction, or the cost of care.⁵ Because CPRs are frequently used to assist in diagnosing or determining appropriate treatment strategies, understanding their impact is quite important. Although many of the examples to this point have focused on diagnostic CPRs, the derivation and validation process remains the same regardless of the type of CPR (ie, diagnostic, prognostic, or interventional). The impact analysis may differ between diagnostic and interventional CPRs on the basis of the nature of care given, but all of the basic principles remain the same. To fully illustrate the derivation, validation, and impact analysis process, the Figure shows the steps taken to create the Canadian C-Spine Rule.^11,24–26

View Figure

• Download as Powerpoint
• Download Figure

Teaching CPRs

Students should understand the nuances of CPR derivation and validation to ensure they can accurately apply them to patients and avoid errors in clinical decisions. For example, if a student did not understand the derivation process and implemented with an active, adolescent patient a CPR that was originally created and validated in a geriatric population, the student could be putting the patient at risk because he or she is using the CPR inappropriately. Furthermore, there are many CPRs that are derived and published before a validity study; thus, students must be able to read and interpret the derivation studies so they can understand how best to use the information. Although CPRs are generally easy to apply, it is important that students and clinicians understand the clinical tools they are using. In the following sections, we will discuss how to incorporate teaching CPRs in a variety of different athletic training courses. We aim to provide ideas to use with students to help them understand all aspects of CPRs. In addition, we will provide ideas on how to help integrate CPRs into the clinical experience by involving preceptors in the educational process. The ideas we present are suggestions or possibilities, and we hope the information and techniques described will help educators and preceptors encourage students to expand their knowledge of CPRs and begin to integrate them into clinical practice. We understand that each academic program and educator has unique nuances, so we aim to provide a variety of suggestions that educators can choose to best fit their course and program progression.

Teaching CPRs in Orthopaedic Assessment Courses

Teaching students about the creation and use of CPRs seems to most naturally fit in orthopaedic assessment courses. Diagnostic CPRs are typically more straightforward in terms of variables assessed and provide students a clear picture of how CPRs can be implemented in practice. We recommend teaching the derivation and use of CPRs in assessment courses along with other evidence-based medicine concepts. McGinn et al¹² suggested educators should start by demonstrating the variability in clinical judgment from clinician to clinician. This can be accomplished by presenting a patient case to the entire group and then asking each individual to write down his or her thoughts regarding the probability the patient has a certain condition. The idea is to then compare the similarities and discrepancies in the answers in order to demonstrate the amount of variance among peers. Helping students understand the original purpose of CPRs and other evidence-based–medicine tools may help them gain a better appreciation of when to apply such tools and how to balance those findings with their own clinical judgment.

Another point of discussion should be determining when CPRs may be useful. Educators could ask students, in groups or individually, to identify a particular injury or condition they see frequently during clinical education and determine whether a CPR for that condition would be helpful. After the condition of interest has been identified, students might be expected to go through the process of defining an outcome variable, creating a list of predictor variables, and sharing their variables with their preceptor to determine whether the preceptor felt they were missing any potential predictor variables. The goal of such an assignment would be to familiarize students with the process of identifying a clinical issue and thinking through the process of creating a CPR. This simple design should allow students to work through the derivation process in a straightforward manner while also integrating the knowledge and expertise of the clinicians with which they work. Additionally, defining outcome and predictor variables may help them develop a better understanding of the conditions they choose.

While teaching students how to create CPRs is important, educators should also be sure to teach students about the CPRs that already exist and how to be able to select and critically appraise those tools for use in the clinical setting. Educators could assign students specific injuries or conditions and ask them to locate at least one related CPR and provide a detailed explanation of the derivation and subsequent validation (when applicable) of the CPR and to provide their references as support. This will give students the opportunity to practice their literature-searching skills, to interpret diagnostic accuracy measurements, and to determine any clinical benefits of implementing the CPR into practice. McGinn et al⁴ suggest educators use a similar approach when teaching students about identifying and selecting different CPRs. They recommended providing students with a patient case and a related CPR and asking them to complete the CPR for the patient case provided. For example, the patient case could be of a soccer player who experienced an ankle injury during a practice session, and the CPR provided might be the Ottawa Ankle Rules. After that activity, McGinn et al⁴ suggest the educator facilitate a discussion on the experience including how the students felt about the CPR and a comparison of the students' results. Researching and learning about existing CPRs will potentially help students better understand the derivation process. Students could also examine existing CPRs to see the differences in how they were validated in clinical settings that may be different from athletic training. In doing so, students might learn how to identify potential pitfalls and benefits of existing CPRs when applied to patient populations they commonly see in their clinical experiences. Finally, students could create 1-page flyers for existing CPRs to hang up in their clinical site. Having a quick 1-page visual reference makes it easier for the student and preceptor to use the CPR with a patient.

Teaching CPRs in General Medicine Courses

Clinical prediction rules outside of orthopaedics can also be quite useful, but it is likely these are less well known than their orthopaedic counterparts by athletic trainers. The CPRs related to the diagnosis of general medical conditions can help athletic trainers have more confidence in their diagnosis or referral decision. These general medical CPRs may find a more natural home in courses centered on general medical conditions and/or pharmacology, which may provide an opportunity to expose students to a variety of nonorthopaedic CPRs and allow them time to become familiar with using them. For example, if a patient reports to the athletic training facility complaining of a sore throat, the Walsh Strep Throat Score^27,28 or Modified Walsh Strep Throat Score²⁷ could help determine the likelihood the patient has a streptococcal infection. Because there are existing CPRs for a variety of general medical conditions (eg, deep-vein thrombosis,²⁹ pulmonary embolism,³⁰ and strep throat^27,28), helping students locate and use those should be a focus of these discussions. These types of activities will help students gain a broader knowledge of the CPRs already in existence as well as provide further practice searching current literature.

A simple activity students could complete when they are doing a nonorthopaedic clinical experience is to interview their preceptor(s) regarding how often they use CPRs in clinical practice. This will give students a general idea of the prevalence of CPRs and allow the students to understand why CPRs are or are not being implemented. They may find their preceptor is unaware of some of the CPRs applicable for that setting, which would provide the students an opportunity to share their knowledge of CPRs with the preceptor. This will encourage collaboration among the students and preceptor and could also improve interprofessional communication. If the preceptor is knowledgeable about CPRs, students could collaborate with the preceptor during their nonorthopaedic clinical experience to create a CPR for a general medical condition they see somewhat frequently. For example, if a student who is completing a clinical experience at a physician's office sees a number of patients with influenza, that student could collaborate with the physician to determine outcome and predictor variables for an influenza CPR. Moreover, students could journal about their experiences working alongside the physician or nurse to help promote a deeper appreciation of interprofessional collaboration.

Teaching CPRs in Therapeutic Interventions Courses

We have discussed CPRs directly related to the diagnosis of various orthopaedic injuries and nonorthopaedic conditions, and although athletic trainers tend to rely primarily on CPRs related to orthopaedic assessment, there are many CPRs that are prognostic or interventional in nature. Prognostic and interventional CPRs can provide valuable information postdiagnosis regarding likely outcomes or potential responses to a specific treatment or combination of treatments.¹ These types of CPRs may help athletic trainers develop more effective treatment plans. In addition to these interventional CPRs, we must also note those CPRs intended to provide information regarding patient prognoses. Like interventional CPRs, prognostic CPRs are less understood and less widely used.

To help students understand the variations in purpose and use of these different types of CPRs, it seems most logical to integrate them into courses in which they can be directly applied rather than simply providing general information about CPRs in one or perhaps two courses. Students may struggle initially to understand the creation and application of CPRs that are nondiagnostic in nature because the predictor and outcome variables are somewhat less straightforward (eg, a tibial fracture versus a positive outcome postrehabilitation). Given that the basic process for creating and validating all CPRs are the same, many of the same activities can be used in these courses that are used in the evaluation courses. Students could be asked to identify a patient case from their clinical site involving a chronic or postsurgical condition and apply either an interventional or prognostic CPR with the assistance of their preceptor. In doing so, students may gain an understanding of how these types of CPRs can be applied with a patient. Students could also identify a patient case and retrospectively apply the CPR. Finally, students could be asked to identify a treatment of interest to them (eg, a modality, manual therapy technique, or rehabilitation program) along with a condition or patient case where their selected treatment is warranted. Students could then create a CPR using their treatment and condition. To develop a better understanding of the validation process, students could be assigned an interventional CPR. Each team could create a mini “study” to validate the CPR including identifying a criterion standard reference variable. Students would be responsible for educating the clinicians at their clinical sites on their CPR and how it is to be applied. Over the course of the clinical experience, students would track the number of cases during which the CPR was used as well as collect data from the reference variable. At the conclusion of the project, students would be asked to compare the results from the CPR and the reference variable.

Teaching CPRs in Organization and Administration Courses

Typically, an organization and administration course allows students to evaluate issues and topics outside of direct patient care and explore the ways in which those things influence the quality of care provided. Therefore, teaching students to examine the more global impact of the CPRs they are implementing is important. It may be easy for students to see the impact of using a CPR on a single patient case, but it can be more difficult to understand how, on a larger scale, using CPRs influences their decisions and practice patterns. Teaching students to use the technology and electronic health records tools available to them to track and analyze their practice may help them become more reflective clinicians and appreciate the widespread impact of the decisions they make each day. This is the central idea of an impact analysis, in that during this evaluation of practice patterns we hope students gain an understanding of the broader impact of using a CPR outside of a single patient case. Furthermore, in such courses we are able to explore the impact of health care informatics and how the use of technology to store, manage, and compile clinical data can influence practice. Asking students to review the electronic health records systems at their clinical sites and identify an outcome variable and predictor variables that are based on their search would be a good way for educators to take students through the CPR creation process from the perspective of a head athletic trainer or facility supervisor.

Using the impact analysis done on the Canadian C-Spine Rule and the fact that it revealed a significant reduction in radiographs ordered in emergency departments after the implementation of the Canadian C-Spine Rule,²⁶ we are able to launch a discussion about how to best determine the effects our decisions have on patient care. Students need to understand that their administrative decisions are not isolated to their clinical practice sites or clinics. Instructors could ask students to identify one CPR in use at their clinical site and track the use and outcomes of that CPR. For example, if a student selected the Ottawa Ankle Rules, that student should track the number of times the CPR was used, what the results of each evaluation were, and the eventual patient outcomes. Educators could then ask students to identify instances in which using the CPR helped them identify a condition they might have misdiagnosed or that influenced the cost of patient care (eg, no advanced imaging was completed due to the findings from the CPR). This activity, which could be done either retrospectively or prospectively depending on the quality of the health records available, may help students gain a better understanding of the impact of incorporating CPRs into their clinical practice. Students should be encouraged to share their findings with the preceptors at the site. This would require students to not only evaluate the CPR's impact but to also provide a summary of the data they compiled to stakeholders, much like they would need to do in their future jobs.

Students could be asked to complete a pre-post intervention “study” using a certain CPR at their own clinical sites. Students could be grouped according to clinical site and asked to select one CPR related to cases they are likely to see throughout their rotation (regardless of CPR type). For example, a group of students may choose to evaluate the Ottawa Ankle Rules due to the high incidence of ankle sprains they see among basketball players. Each group would then create their own mini-impact analysis. In a final report, they would need to provide background information on the CPR, particularly on its validation, as well as background information on the clinical site in relation to the condition being examined. Each group must specifically define their target population and the ways in which they would educate the other clinicians at their site about using the CPR. Finally, they would conduct the mini-impact analysis and give a report of their results in class and to their preceptors.

An additional activity that might be included is asking students, in groups, to create 35- to 45-minute educational sessions on a CPR of their choosing. Depending on availability and number of students and/or groups giving presentations, students could present their findings to all the program's students, faculty, and preceptors, or other local health care providers. Students should be given some measure of creative license. However, presentations likely should include background information on the condition of interest, the CPR, and all validation done on that CPR. Presentations should also include detailed information on how incorporating that CPR into clinical practice may influence interactions between athletic trainers and other health care providers. These activities are designed to help educators connect students with information on CPRs as well as gain a better understanding of how using CPRs can influence more global practice patterns.