Evaluating Evidence-Informed Clinical Reasoning Proficiency in Oral Practical Examinations

INTRODUCTION

Clinical reasoning is the nuanced, yet advanced, cognitive process whereby health care practitioners quickly and safely formulate accurate diagnoses for complex patient problems. Central to this cognitive mechanism is the ability to consider and rule out relevant and competing differential diagnoses in order to act upon subsequent management decisions, and to develop safe and effective plans of care for patients.¹ Athletic training is no different than its medical and health care contemporaries—both students and practitioners need to develop and display effective clinical reasoning skills in the assessment of injury and illness in order to provide evidence-based functional outcomes. In clinical practice, a sound and accurate diagnosis (Dx) is indeed the first step towards developing favorable and efficient outcomes; thus, the thinking process that initiates each and every patient encounter must be sound and sophisticated.

Whereas novice clinicians favor the more intuitive, hypothetico-deductive approach to reasoning that's characterized by the formation of multiple hypotheses followed by prolonged, unorganized, and time-intensive evaluation procedures, more experienced practitioners display more streamlined and accurate thinking by using case pattern presentation (CPP) as their primary mode of thinking through complex patient problems.¹ It is central to the ability to formulate, store, and recall specific CPPs that the capable clinician be able to readily identify key features (KFs) that fit within a known and experienced pattern, while equally recognizing features that don't fit known or previously experienced patterns—which lead to new case patterns, in a cyclical fashion. Akin to an experienced criminal detective, an experienced clinician is able to quickly and adeptly discern relevant from irrelevant data points, and, better yet, accurately connect the dots between various features (signs, symptoms, mechanism of injury, physical exam results, etc) presented before him or her.

Further, a critical component of effective clinical reasoning is the ability to properly select and interpret various conclusions from the mounting quantity of evidence-based medicine (EBM), particularly as the evidence pertains to the diagnostic accuracy and predictability of various special tests and physical examination procedures.² It's essential for both developing and practicing clinicians to learn to judiciously and effectively integrate the best evidence available for the diagnostic utility of each clinical test they are taught if they intend to develop the level of competence needed to (1) accurately diagnose illness and injury and (2) subsequently implement effective and efficient treatment plans. To reach this level of evidence-informed clinical competence, students and clinicians alike must be made aware of the diagnostic properties of tests and measures and know which of them have clinical utility, and which of them do not—or at least which of them are limited.³ Specifically, practicing clinicians need to be adept at wading through the various and intimidating statistics behind the plethora of special tests and other physical examination procedures put forth in academic texts, professional journals, and educational programs, and, at some point in their development, they need to know which have usable properties associated with them and which do not.⁴ The sometimes daunting numbers behind sensitivity, specificity, likelihood ratios, predictive values, Quality Assessment of Diagnostic Accuracy Studies, and utility scores must become KFs in the clinical mind; they must become a natural and usable part of the clinical decision-making process if sound diagnoses are to be made with accuracy and efficiency.

As clinical reasoning and evidence-informed practice are the centerpieces of our institution's athletic training education philosophy, we have intentionally sought mechanisms for teaching and evaluating the aptitude of our students to develop and appreciate this complex and nuanced faculty of the mind. For the oral practical examinations in our Assessment of Injuries laboratory class, we have integrated clinical reasoning with evidence-based practice (EBP) by developing evaluation models that are capable of addressing the critical interconnections between the 2 concepts of mind and practice. As this class is taught to our second-semester sophomore athletic training majors, this pedagogical strategy represents our students' initial exposure to the integration of clinical reasoning and diagnostic accuracy. From a programmatic perspective, this initiative is a central component of our attempts to weave both clinical reasoning and EBP throughout our entire curriculum, both didactically and clinically. As extensive reviews of clinical reasoning and the components of diagnostic accuracy are available elsewhere, the point herein is not to cover those concepts extensively, but rather to share 1 example of how our program attempts to introduce and assess these higher order thinking skills in our more novice students.

ORAL PRACTICAL EXAMINATIONS

The assessment-based oral practical examinations we have developed are designed to challenge and assess our students' ability to understand and apply both clinical reasoning and EBP in a mock-clinical context. These examinations were born out of, and represent a modification of, the significant work done in medical education on the use of KF examinations for assessing clinical decision-making skills by Page et al.⁵ In summary, the patient-based scenarios are role played by seasoned clinicians who provide a series of “macro evaluations” for the student to address. Each final oral practical examination is made up of a collection of small scenarios that are either presented as a definitive Dx, or a small list of KFs from known orthopaedic CPPs. When given an actual Dx to work from, students are in effect working backwards from that Dx, and are asked to demonstrate mastery of certain components of an evaluation relative to that Dx, a macro evaluation in that it is not a complete A to Z evaluation of the condition (see Figure 1). Because the Dx is known in these sections of the overall exam, we are intending to address what might be called lower-level clinical reasoning, which in reality is befitting of the novice status of our younger students.

View Figure

• Download as Powerpoint
• Download Figure

In contrast, presenting only select KFs of known CPPs and asking students to work through certain components of an evaluation requires higher-level clinical reasoning because it forces students to immediately generate a differential Dx before proceeding with their physical examination (see Figure 2). This more sophisticated thinking requirement is more typical of advanced, experienced clinicians and is representative of what Bordage⁶ calls elaborated knowledge, by requiring students to more clearly define the patient problem early on in the process and to demonstrate a deeper meaning of the signs/symptoms and other data presented (as KFs). In elaborated knowledge, signs and symptoms are never totally understood in isolation, but rather are considered in relation to the position within the total set of signs and symptoms being presented.

View Figure

• Download as Powerpoint
• Download Figure

In the final examination version, each student will get a combination of macro evaluations—1 or 2 known diagnoses to evaluate, and 1 or 2 KFs of a known CPP—and there will be a mix of simple and complex cases to address in multiple joints for each student. After performing the required macro evaluation steps (some ask for a history, for an observation, etc) and other contextually relevant physical exam steps (relevant palpation, range of motion [ROM], manual muscle testing [MMT], etc), students are then required to appropriately pick the most fitting and accurate diagnostic tests to fit the CPP being portrayed by the mock patient. Figure 1 is an example of a simple lower extremity known Dx macro evaluation, and Figure 2 is an example of a more complex lower extremity KF presentation; each represents 1 part of an overall final examination. In designing these sorts of evaluations, we are attempting to foster more streamlined, organized, and clinically relevant assessment behaviors and habits in our novice students, and intending to prevent them from the age-old habit of doing every special test they know for the joint under investigation. We are directly intending to foster, develop, and assess a level of elaborated knowledge in our students by getting them to clearly define the patient problems that we have covered in class via the use of differential Dx tables, and we are equally trying to avoid the “jaundiced textbook” phenomenon, whereby students can be seen highlighting entire book chapters and then attempting to do everything on a subsequent practical evaluation.⁶ Throughout the practical exam, students must not only match the appropriate special/functional tests, etc, to the definitive Dx given or KFs provided (thus demonstrating their mastery of the deeper linkages between the various signs and symptoms that are central to the ability to compare and contrast among competing diagnoses), but also choose and explain how useful the chosen test(s) are for the differential Dx process being entertained, relative to the best evidence (sensitivity/specificity, likelihood ratios, etc) available (thus demonstrating their understanding of how and when to use “best evidence available”).

For example, if the mock patient presented KFs such as a valgus rotation mechanism of injury, reportedly hearing/feeling a loud pop and a sensation of the knee shifting at the moment of injury, the successful student would develop a differential Dx of sprains to the medial collateral ligament, the anterior cruciate ligament (ACL), and maybe even the medial meniscus, and perhaps even a subluxed/dislocated patella. Because the differential Dx includes a ruptured ACL, the student should first choose (and perform) the Lachman test as the best test to confirm and/or rule out this Dx, based on its documented utility score of 1.⁷ Priority points are given for performing the Lachman test as the best choice to confirm/rule out the ACL damage, and then lower point values are given for students who choose to do the anterior drawer as the best choice (with utility score of 2, based on lower specificity and sensitivity values reported in the literature). Thus, a student who accurately formulated an ACL sprain as his/her number 1 Dx, and chose to execute and explain the Lachman test first, followed by the anterior drawer, would score more points than a student who formulated the same Dx, yet chose to do the anterior drawer first followed by the Lachman test. Fundamentally, both students would have demonstrated sound clinical reasoning in recognizing the KFs presented and formulating a likely differential for the given scenario, and both students would have chosen applicable special tests specific for the decision making for the ACL element of the Dx, but one student would have demonstrated a better mastery of the evidence in choosing the Lachman as the best test for the Dx under consideration.

SCORING AND FEEDBACK

In general, the scoring/grading of each section is somewhat fluid in that each instructor/mock patient is allowed some latitude to “go where the student goes” in the evaluation (given the clinical context). Not only is this element of grading more or less required because of the time-intensive nature of these exams and limited personnel, but also it is consistent with prior research done on KF examinations. Page et al⁵ point out that flexible scoring is required to accommodate the complexity and configurations of actions often required in the resolution of real-world clinical problems. We believe in the very real notion of clinical and learning context, and that not all things can be controlled for in teaching and learning environments—especially in the realm of clinical learning and the impact and role of clinical experience (from both the instructor doing the grading, and the student attempting to demonstrate mastery). Thus, we follow the recommendations put forth by Page et al⁵ by allowing our various instructors to grade students as they see fit and to provide specific and constructive feedback that is contextually relevant to each student at the same time. If you look at the 2 examples provided herein (Figures 1 and 2), you will also notice differentiated point values for various items. In keeping with the blueprint put forth by Page et al,⁵ we also realize that not all steps in the resolution of problem are of equal importance, and so we subscribe to their idea that better time can be spent by focusing on what we feel are the critical steps of certain evaluations, while being respectful of the clinical context element at the same time. To help the reader better use and understand our macro exams, we offer a few guidelines for assistance while fully realizing that programmatic differences and pedagogical style will largely influence how one chooses to implement what we are offering:

1. Points are not deducted when students do something not listed on the evaluation form. For example, if a student palpates medial bony structures and performs a valgus stress test for the lateral knee pain scenario in Figure 2, the student is not penalized any points. The student merely doesn't get points added, and feedback is given that it is not necessarily wrong to do those things in this scenario, but they can distract from the important steps in an effective and streamlined evaluation of this nature.

2. There is room to award points for a student choosing to perform something not listed on the evaluation form, something not thought of. For example, if the student wanted to evaluate the patient's gait in order to see what the source of the patient's pain/dysfunction was in the scenario in Figure 1, that would be an appropriate strategy, and so the student would get rewarded. This is an example of the flexible scoring we refer to.

3. Each instructor can award from 0 to 3 points for the execution of tests and evaluation skills (ROM, MMT, etc) based on the instructor's assessment of the student's performance. Before this oral practical scenario, all of our students will have been previously evaluated on each subskill with a more discerning and detailed rubric and in multiple formats, so we will have previously addressed the details of the psychomotor execution and will be fully comfortable with a more relaxed scoring mechanism that focuses on context and decision making at this point.

4. Differentiated point values and what's included as required in each scenario are based on various points of emphasis in our program, in both what we teach and how we teach it, and are based on our concept of differential Dx making and the use of KFs. Other instructors may choose to modify the content or point values, or to prioritize other elements of evaluation strategies relative to particular joints and/or diagnoses.

SUMMARY

Using these exam formats forces novice students to match various KFs (eg, history, quality of pain) of either a mock CPP or a definitive Dx, and then to select and demonstrate the best steps for evaluating and decision making, thus requiring them to demonstrate proficiency in cognitive processing and skill execution as well as the application and interpretation of those skills. The purpose of this short paper was to present 1 method for evaluating the ability of students to properly select, perform, and interconnect various physical examination skills and the relevant evidence available, all as interconnected components of their developing clinical reasoning aptitude.

Since the introduction of the fifth edition of the Athletic Training Education Competencies,⁸ the integration of EBM into established athletic training curriculums continues to pose several challenges for students, educators, and clinical preceptors alike.^9,10 The 14 new EBM competencies consist of fairly advanced concepts and skills that must be taught, learned, and most importantly practiced in all educational programs. Essentially, this requires athletic training faculty members to develop and implement effective pedagogical strategies for turning EBM into EBP in neophyte clinicians. To complicate matters, many practicing clinicians were not formally taught the principles of EBM during their educational preparation, and many have limited resources and time for learning and doing EBM on a regular basis, forcing ATEP faculty members and administrators to make the time to learn the concepts of EBM and then become confident enough to teach this information to students and their clinical educators simultaneously.⁹ Assuring that students understand these concepts and how (and when) to apply relevant elements of EBM in clinical settings and contexts can be difficult for educators unfamiliar with the idea and principles of EBM. Additionally, faculty members face many challenges keeping up with new, emerging evidence that is constantly evolving and finding effective ways to incorporate new evidence and ideas into their pedagogy and across their curriculum.

Beyond the challenges facing athletic training educators, students also face considerable challenges absorbing these new concepts and learning to apply these relatively complicated processes to their clinical experiences in meaningful ways. Regarding diagnostic reasoning, athletic training students must take the evidence being taught to them, such as sensitivity and specificity statistics, or utility scores, and determine what makes a test valuable, and then figure out exactly what to do with the information in the clinical context. For example, although a test may have a low utility score, the sensitivity or specificity number may still make the test clinically relevant in a given context. As educators, we must constantly help students digest the profusion of information and form clinical decisions that position the patient at the center of the process and lead to sound and safe clinical decisions. This process is challenging for seasoned clinicians alone, never mind undergraduate, entry-level students who are simultaneously learning and developing the bioscientific knowledge basis (anatomy, physiology, pathophysiology, biomechanics, etc) and psychomotor skill sets required to understand and perform athletic training.

We have been using these formats for the last 2 years with pleasing success while also knowing that, like most pedagogical approaches, this is not yet a perfect system; we continue to modify and amend our scenarios, the formatting, and our grading criteria/procedures to better fit our needs and intent. Student response has been overwhelmingly positive, as students report liking the real clinical context and concerted emphasis on thinking and doing in a clinical context. They also appreciate the instant and direct feedback we provide not only on their skill execution, but more importantly on their thinking—why they chose to perform certain tests and physical exam procedures. Developing, conducting, and scoring these types of evaluations is indeed time- and energy-intensive on our part, yet to date we are happy not only with what the students tell us about the challenges associated with this thinking-based microexam format, but also with what these exams tell us about our teaching and direct attempts to integrate our curriculum with clinical reasoning and EBP. We feel strongly that it has been an instrumental step towards our efforts to get our students to move towards more elaborated knowledge as they begin to embark on the more meaningful and challenging clinical experiences that lie ahead.