Standardized Patients Provide a Reliable Assessment of Athletic Training Students' Clinical Skills
Context: Providing students reliable objective feedback regarding their clinical performance is of great value for ongoing clinical skill assessment. Since a standardized patient (SP) is trained to consistently portray the case, students can be assessed and receive immediate feedback within the same clinical encounter; however, no research, to our knowledge, has documented the reliability of the SP at assessing student performance.
Objective: To determine if SPs provide a reliable assessment of athletic training students' performance in obtaining a patient history and completing a physical examination relative to athletic training faculty.
Design: Reliability study.
Setting: Athletic training simulation lab.
Patients or Other Participants: Two SPs and 2 athletic training faculty assessed 35 students (n = 20 junior; n = 15 senior) in athletic training cohorts from a public liberal arts institution in southeast United States.
Intervention(s): Athletic training students completed 2 SP encounters per semester throughout 1 academic year in the athletic training program, totaling 4 SP encounters.
Main Outcome Measure(s): After each SP encounter, athletic training faculty and SPs completed the same clinical performance checklist developed specifically for each encounter. The checklist included yes/no items related to obtaining a patient history (10–12 items each) and completing a physical examination (12–15 items each). For each SP encounter, composite scores were computed for both history and physical examination items from the athletic training faculty and SPs. Intraclass correlation coefficients (ICC) determined interrater reliability between athletic training faculty and SPs for history and physical exam items.
Results: Reliability coefficients between the SP and athletic training faculty indicated fair to strong agreement for most history and physical examination items. Significance was found for history items in the cervical spine emergency (ICC = 0.671, P = .002), knee (ICC = 0.696, P = .003), low back (ICC = 0.622, P = .002), concussion (ICC = 0.764, P = .004), general medical (ICC = 0.571, P = .008), and psychosocial intervention (ICC = 0.572, P = .008) encounters. The reliability coefficients were significant regarding physical exam items for the cervical spine emergency (ICC = 0.588, P = .01), low back (ICC = 0.766, P > .001), concussion (ICC = 0.792, P = .001), and general medical (ICC = 0.878, P > .001) encounters.
Conclusions: Overall, the SPs provided a reliable assessment of the athletic training students' clinical performance for obtaining a patient history and completing a physical examination. Given these results, devoting additional time during SP training should increase the reliability of the SP.
INTRODUCTION
A standardized patient (SP) encounter is a real-person simulation that provides health professionals and students with a dynamic, patient care encounter to practice new skills or demonstrate clinical competence.1 Traditional clinical experiences are not sufficient to educate students to perform psychomotor skills competently, even for relatively straightforward patient problems (eg, psychosocial intervention and referral).2 The primary benefit of the SP encounter is that it maximizes patient safety by removing the actual patient while still providing a nonthreatening environment for training or assessment activities.3,4 Standardized patients have been used in athletic training education5–7 to engage students in real-time patient-centered encounters. Previous researchers7 have characterized SP encounters as realistic and worthwhile and have a positive impact on the athletic training student's confidence in future clinical encounters.5,6,8 These types of encounters are important in the education of athletic training students, as patient-centered care (ie, care that is respectful of and responsive to individual patient preferences, needs, and values, and ensuring that patient values guide all clinical decisions9) is a core dimension of quality health care.10
Providing students objective feedback regarding their clinical performance based on a reliable assessment is of great value for ongoing performance assessment.11 Standardized patients provide an objective, standardized, and consistent means for assessing the clinical performance of multiple students.2 Therefore, the purpose of this investigation was to determine if SPs provide a reliable means of assessing an athletic training student's performance in obtaining a patient history and completing a physical examination. It is difficult to find reliable measures to assess students' clinical performance.11
METHODS
This manuscript is part of a larger study5 examining SPs learning outcomes (ie, confidence, ability to obtain a patient history, ability to complete a physical examination, and interpersonal skills) in the professional education of athletic training students.
Participants
A total of 16 individuals were recruited and trained to portray a patient and assess athletic training students' clinical performance throughout the evaluation timeframe. Individuals portraying the SPs were recruited from the theater department to reduce the chances of familiarity with athletic training students being assessed. The individuals portraying the SP were not recruited to portray more than 1 SP role throughout the academic year. In addition, the researchers (2 athletic training faculty) assessed clinical performance during each of the athletic training students' 8 SP encounters.
Procedures
Approval from the institutional review board was obtained prior to beginning the study, and students consented for class data to be used for purposes of the investigation. The SP encounters were completed as a component of the athletic training students' respective clinical education courses. In each academic semester (both fall and spring semesters), both junior-level and senior-level athletic training students completed 2 independent SP encounters. See Table 1 for a matrix of SP encounters. Each of the SP encounters were completed during week 5 and week 10 of both academic semesters. All SP encounters were video recorded.
Each athletic training student was evaluated independently by the SP using a clinical performance checklist. Immediately following the SP encounter, the SP completed the clinical performance checklist for each student. Because the encounters were recorded, athletic training faculty independently evaluated each student from viewing videos (which allowed for the ability to pause/rewind) rather than scoring students during the live encounter. All faculty evaluations were completed within 1 week of each student's SP encounter. Both athletic training faculty completed the same clinical performance checklist that the SP completed for each student from both junior and senior cohorts, providing 3 independent assessments of each athletic training student's clinical skills regarding each of the 8 SP encounters.
Training the Standardized Patient
Training for each of the SP cases included a 60-minute initial training session, where information of the case was presented to the individual portraying the SP. This included current and past medical history, social history, family history, and any affect needed for the case (ie, dress/appearance, psychological mood/state). During the initial training sessions, individuals were provided opportunities to ask questions to ensure their understanding of the case they were portraying. We found it important to never tell the SP their diagnosis, since our emphasis is on the process, not whether students get a correct diagnosis. Initial training also included reviewing assessment methods regarding how to evaluate each athletic training student at the conclusion of the encounter. Each SP reviewed the clinical performance checklist for the case. Specifically, a focus was placed on criteria for successful completion of the item (ie, correct hand placement for a selective tissue assessment, intensity/strength of the palpation). Additionally, each SP was instructed on how to provide feedback to students during the encounter as certain questions were asked. We found it important to stress that, if students ask something we did not discuss, typically a question deemed not important to the overall authenticity of the case, the SP was instructed to answer how she/he would in a natural setting. Each SP also completed a review session 30 minutes prior to student evaluations.
To serve as a baseline for comparison, reliability between the athletic training faculty was established. Reliability measures regarding the athletic training faculty's ability to assess students complete a physical examination were statistically significant for the cervical spine emergency (intraclass correlation coefficients [ICC] = 0.752, P < .01), knee (ICC = 0.704, P = .001), low back (ICC = 0.824, P > .001), concussion (ICC = 0.528, P = .003), and general medical (ICC = 0.952, P > .001) encounters.
Instrumentation
Objective dichotomous (yes/no) checklists were used to enhance interrater reliability and increase the number of skills that could be evaluated in a brief period of time.12 The checklists were developed by the researchers and included skills7,13–20 to obtain a patient history and complete a physical examination specific to the SP case. Each of the clinical performance checklists consisted of 10–15 items related to obtaining a detailed patient history and 8–15 items related to completing a physical examination. The development of each SP case and how individuals portraying the SPs were trained have been previously described.5 All responses were captured electronically via SurveyMonkey (Palo Alto, CA).
Cronbach α determined internal consistency for obtaining a patient history and physical examination items, with an α coefficient between 0.412–0.690 and 0.706–0.779, respectively.
Data Analysis
Descriptive statistics were computed for all clinical performance checklist items. Items on the clinical performance checklist were scored yes/no for each specific checklist item. For each clinical performance checklist, composite scores were computed for both obtaining a patient history and completing a physical examination. Items from the athletic training faculty were compared with scores from the SPs. Additionally, composite scores from each individual athletic training faculty were compared regarding obtaining a patient history and completing a physical examination. Two-way mixed model ICCs for absolute agreement and 95% confidence intervals (CIs) determined interrater reliability between the composite of athletic training faculty and SPs for history and physical exam items. An α level of .05 was used for all analyses. Data analyses were performed using IBM SPSS (version 21.0; SPSS Inc, Chicago, IL).
RESULTS
Overall, reliability coefficients between the SP and athletic training faculty indicated fair to strong agreement for history and physical examination items, indicating that the SPs provided a reliable assessment of the athletic training students' clinical skills. Intraclass correlation coefficient scores were interpreted as follows: 0–0.2 poor agreement; 0.3–0.4 fair agreement; 0.5–0.6 moderate agreement; 0.7–0.8 strong agreement; and >0.8 almost perfect agreement.21
Obtaining Detailed Patient History Items
Standardized patients provided a reliable assessment of each student's ability to obtain a detailed patient history during most patient encounters. Interrater reliability (ICC values) ranged from 0.447 (fair agreement) to 0.764 (strong agreement) for obtaining a detailed history, indicating a fair to high level of agreement between how the SP and athletic training faculty assessed each student's clinical performance. Table 2 details ICC values and 95% CIs comparing SP and athletic training faculty scores for each SP encounter regarding obtaining a patient history.
Specifically, ICC values were statistically significant between the SP and athletic training faculty regarding the students' abilities to obtain a patient history on the cervical spine emergency, knee, low back, concussion, general medical, and psychosocial intervention encounters. These ICC values indicated moderate level of agreement for the cervical spine emergency (ICC = 0.671, P = .002), knee (ICC = 0.696, P = .003), low back (ICC = 0.622, P = .002), general medical (ICC = 0.571, P = .008), and psychosocial intervention (ICC = 0.572, P = .008) encounters; and high level of agreement for the concussion (ICC = 0.764, P = .004) encounter. Because the bounds of the 95% CI for the shoulder encounter crossed the null, the data indicating fair level of agreement are not presented.
Athletic training faculty assessments were compared to ensure that athletic training faculty provided a reliable assessment of each student's skills regarding obtaining a patient history. Intraclass correlation coefficient values ranged from 0.477 (fair agreement) to 0.838 (strong agreement) for obtaining a detailed history. Table 3 details ICC values and 95% CIs comparing athletic training faculty scores for each SP encounter regarding obtaining a patient history.
Completing a Physical Examination Items
Interrater reliability (ICC values) ranged from 0.451 (fair agreement) to 0.878 (strong agreement) for completing a physical examination, indicating a fair to strong degree of agreement in how the SP and athletic training faculty assessed each student's clinical performance. Table 4 details ICC values and 95% CIs comparing SP and athletic training faculty scores for each SP encounter regarding completing a physical examination.
Specifically, ICC values were statistically significant between the SP and athletic training faculty regarding the students' abilities to complete a physical examination for the cervical spine emergency, low back, concussion, and general medical encounters. These ICC values indicated fair level of agreement for the cervical spine emergency encounters (ICC = 0.588, P = .01); and high level of agreement for the low back (ICC = 0.766, P > .001), concussion (ICC = 0.792, P = .001), and general medical (ICC = 0.878, P > .001) encounters. Because the bounds of the 95% CI for the shoulder encounter cross the null, the data indicating fair level of agreement are not presented.
Athletic training faculty's assessments were compared to ensure that athletic training faculty provided a reliable assessment of each student's skills regarding completing a physical examination. Intraclass correlation coefficient values ranged from 0.528 (moderate agreement) to 0.952 (strong agreement) for completing a physical examination, indicating a moderate to high level of agreement between how the athletic training faculty assessed each student's clinical performance. Table 5 details ICC values and 95% CIs comparing athletic training faculty scores for each SP encounter regarding completing a physical examination.
DISCUSSION
Standardized patients provide real-time patient encounters that add depth and breadth to the array of teaching and evaluation techniques used in health care.1 While the potential exists for variation in the portrayal of cases within and between SPs, previous researchers22 have reported that SPs are more than 90% accurate in portraying details of the case. Standardized patients provide students with real-time patient encounters necessary for professional practice that they may not be exposed to during clinical education. Our purpose was to determine if SPs provide a reliable means of assessing an athletic training student's performance in obtaining a patient history and completing a physical examination compared with athletic training faculty's assessment. To ensure accurate portrayal and assessment of the cases, we provided stock responses to certain common questions students were likely to ask,23 as well as feedback to SPs between student examinees.
Petrusa2 reported that SP-based examinations are arguably the most extensively researched educational innovation in medical education. The recent increase in the research of SPs in athletic training education is a testimony to the technique and to the benefits that have long been valued by medical education. In the present study, the SP encounters incorporated through the curriculum helped bridge the gap between practice areas where athletic training students reported few opportunities for real-time patient encounters (ie, cervical spine emergency, psychosocial intervention). We also included those encounters students are likely to have seen in clinical practice (eg, low back, knee).
Medical education research outcomes document that SPs are comparable to instructor evaluations and provide reliable scoring of students and practitioners.1 In the present study, we found that SPs provided a reliable assessment of athletic training students' ability to obtain a patient history (moderate or high level of agreement during 6 of 8 SP encounters) and complete a physical examination (moderate or high level of agreement during 4 of 6 SP encounters). Researchers in nursing education1 reported ICCs that revealed moderate to substantial (strong) agreement between the SP and second rater team scores. Thus, overall, SPs do provide a reliable assessment of students' clinical skills. However, it should be noted that some encounters (ie, psychosocial intervention, general medical, and nutrition encounters regarding patient history; cervical spine encounter regarding physical examination) had lower bound CI intervals of poor reliability. These results indicate the need for additional training for SPs portraying those specific encounters, but also the need for additional research regarding the reliability of SPs in evaluating athletic training students' clinical performance.
In the present investigation, the SPs completed the clinical performance checklist immediately after the athletic training student completed the encounter, whereas athletic training faculty completed the checklists from a recording of the SP encounter. Researchers11 have found assessment via a set of grading checklists used by nonexpert raters (ie, SPs) to have a high degree of reliability (moderate to strong level of agreement) that is not affected by whether students are scored during or after the patient encounter. However, it should be noted that Stillman et al24 found low agreement when faculty rated students' clinical skills from videotaped encounters. This finding may indicate why levels of agreement between the athletic training faculty were lower during certain SP encounters, insofar as it was often difficult to determine students' abilities to perform certain clinical skills from a video recording (eg, camera angle was not sufficient to determine proper skill demonstration, student stood between the SP and the camera). This could also provide some insights as to why a high level of agreement did not exist between the SP and athletic training faculty for all encounters.
Providing objective feedback to students regarding their clinical performance is of great value for ongoing clinical skill assessment and lifelong learning.11 We found that SPs provided athletic training students with objective feedback via the checklist and individual comments regarding clinical skill performance from the perspective of the patient. This feedback gives specific examples of where students can improve. Through the current investigation, we found this feedback to be consistent with the assessment of the student's performance as indicated within the checklist.
Previous researchers25 offered that shorter checklists hold numerous advantages, including higher levels of agreement between the SP and other raters due to fewer items. Interestingly, our findings only partially support this notion. Instances where fair or moderate levels of agreement between SP and athletic training faculty did not occur were consistent during encounters with larger numbers of checklist items. This may suggest cognitive overload on the SP, indicating an area where further research is needed. Levels of agreement were low regarding obtaining a patient history during the nutrition encounter (with 15 checklist items). Additionally, several encounters yielded lower bound CI values or poor reliability (0.125) or CIs that crossed the null (eg, shoulder lower bound −0.044, upper bound 0.765), which indicates that further research is necessary to understand why such a difference exists in how the SP and athletic training faculty assessed the students' clinical performance. The results of this study indicated that additional time was needed during initial and follow-up SP training regarding clinical performance checklists items.
Limitations and Future Research
This study provides preliminary data that can be compared across multiple years to better determine trends in the reliability of SPs to accurately assess students' clinical performance. One limitation is the use of students within 1 athletic training program. While the present investigation used 2 cohorts of athletic training students, the data revealed that SPs do provide a reliable assessment of students' clinical skills. This research adds to the growing body of literature in athletic training education to support the use of SPs in the professional education of athletic training students. Additionally, dichotomous checklists (yes/no) were used to assess students' clinical skills. Further research could examine the reliability of SPs using a global rating scale/checklist to assess students' clinical skills during patient encounters.
The present study demonstrated the necessity for providing proper training for the individual playing the role of the SP. We found fair to strong levels of agreement between athletic training faculty and SPs. While the lower levels of agreement (fair and moderate) expose areas for further research, this finding also stresses the importance of providing adequate initial and follow-up training sessions for the SP. These results also suggest that further exploration be considered to understand why SPs and athletic training faculty provide reliable assessment of athletic training students' clinical skills during certain patient encounters but not others. Specifically, why did the SP and athletic training faculty have low level of agreement when assessing students' performance for the nutrition encounter (which emphasized obtaining a patient history), yet had high levels of agreement regarding students' performance during the concussion encounter (which emphasized completing a physical examination and providing patient education)?
Since the SPs provided reliable assessment of the athletic training students' clinical skills, students should reflect on the feedback given by the SPs for future patient encounters. Future research should examine how students use the feedback provided by the SP in modifying clinical practice behaviors. Additionally, future research should also include larger groups of athletic training students, including not only larger cohorts of athletic training students, but also multiple institutions for comparison. Because SPs provided reliable assessments of athletic training students' clinical skills, further research should explore the use of SPs in the context of continuous professional development.2
CONCLUSIONS
Standardized patients provide athletic training students an opportunity to complete real-time patient encounters in a nonthreatening environment that allows for their direct application of knowledge and skill. Because the SP is purposefully trained, they are able to consistently portray the signs, symptoms, and general affect of a patient. The authenticity of the encounter allowed students to interact with the SP in a manner similar to how they interact with a patient during clinical experiences. Overall, the SPs provided a reliable assessment of the athletic training students' clinical performance for obtaining a patient history and completing a physical examination. Devoting additional time during SP training should increase the reliability of clinical performance assessment during the SP encounter.
Contributor Notes
Dr Armstrong is currently an Associate Professor in the Department of Applied Medicine and Rehabilitation at Indiana State University, serving as the Program Director of the Professional Athletic Training Program. Please address all correspondence to Kirk J. Armstrong, EdD, Department of Applied Medicine and Rehabilitation, Indiana State University, 567 North 5th Street, Sycamore Center for Wellness Room 262, Terre Haute, IN 47809. kirk.armstrong@indstate.edu.