Peer Assessment of Clinical Skills and Professional Behaviors Among Undergraduate Athletic Training Students
Context: Peer assessment is widely used in medical education as a formative evaluation and preparatory tool for students. Athletic training students learn similar knowledge, skills, and affective traits as medical students. Peer assessment has been widely studied with beneficial results in medical education, yet athletic training education has thus far produced only 2 studies that address the use of peer assessment with athletic training students.
Objective: To identify whether undergraduate athletic training students accurately and reliably assess their peers on clinical skills and professional behaviors.
Design: Quasi-experimental between-groups and within-group.
Setting: Medical exam office.
Patients or Other Participants: Junior and senior athletic training students, classroom faculty and clinical preceptors (instructors) from the same Commission on Accreditation of Athletic Training Education athletic training program.
Intervention(s): One instructor and 2 students concurrently performed an assessment in real-time of 5 clinical skills performed by a junior or senior student. Clinical skills demonstrated were the Biceps Femoris Manual Muscle Test (BFMMT), Lachman Test, Kleiger Test, Noble's Compression Test, and Thompson Test.
Main Outcome Measure(s): Each student group's scores were compared with instructor group scores to determine the accuracy of the assessment. Each student group's scores were compared within-group to determine reliability of student scores. Accuracy and reliability of clinical skills were measured using the Cohen κ coefficient. A weighted Cohen κ coefficient was used for professional behavior measures.
Results: Senior students were accurate (P < .05) for all clinical skills and professional behaviors and reliable (P < .05) for the BFMMT, Thompson Test, and professional behaviors. Junior students were accurate (P < .05) for all clinical skills except the Lachman Test and reliable (P < .05) for the BFMMT and Noble's Compression Test.
Conclusions: Students are able to assess the clinical skills of their peers with the same consistency as instructors during a live skills demonstration. Year in school may affect ability to assess professional behaviors.
Medical and health care professionals use peer assessment as a valuable method to share knowledge, evaluate performance, and foster professional growth.1–3 Peer assessment nurtures professional growth in students before they enter the workforce.4 It may help develop the self-assessment skills students need to judge their own abilities when working as independent health care practitioners.2 Peer assessment as a formative evaluation tool has been widely studied, with beneficial results in medical education,3,4 yet only 2 currently published studies address the use of peer assessment in athletic training education.5,6
Medical and athletic training programs require graduates to possess and demonstrate knowledge, skills, and affective traits suitable for practice in a modern health care setting, including (1) collaboration with other health care providers; (2) performance of physical examinations to establish diagnoses and develop treatment plans; (3) emergency care for life-threatening conditions; (4) disease prevention through health and wellness promotion; (5) demonstration of professional behaviors and ethical standards; and (6) effective communication with patients and families.7–11 Due to these similarities, students of both disciplines can see similar outcomes from peer assessment. Therefore, athletic training education can benefit from performing peer-assessment research similar to the research performed in medical education.
Early peer-assessment studies in athletic training education show promise for the incorporation of peer assessment in the discipline.5,6 Entry-level graduate students were accurate in the peer assessment of clinical skills using recorded video demonstrations. These students were not reliable in peer-assessment skills during one-time measurement, but their reliability improved as the number of treatments increased.5 In medical education, fourth-year and fifth-year medical students had moderate levels of agreement with faculty on grading objective structured clinical examinations.12 A significant correlation was found between peer ratings and traditional assessment measures, including faculty rating of clinical skills and professionalism, during a medical clerkship.13
There is some evidence to suggest that students tend to be more lenient in the grading of peers than instructors. Machado et al14 compared self, peer, and instructor grades of medical students over the course of 7 semesters and determined that students' self and peer grades were significantly higher than instructor grades every semester and over the course of the study. Also, second-year medical students were significantly more lenient in grading their peers' medical history–taking skills than instructors.2
Using the current literature for context, this study sought to identify whether undergraduate athletic training students accurately and reliably assess their peers on clinical skills and professional behaviors. Investigation into peer assessment in athletic training students can determine peer assessment's use as a valuable evaluation tool in athletic training education, as has been established in medical education.3,4,15
METHODS
Design
This quasi-experimental study used between-groups and within-group measures. Interrater agreement was measured with the Cohen κ coefficient for clinical skills and linear weighted Cohen κ (κω) coefficient for professional behaviors. The accuracy measures compared student (junior or senior) group ratings with instructor group ratings and the reliability measures compared only student (junior or senior) group ratings.
Participants
This study used a convenience sample from an accredited undergraduate athletic training program in the mid-Atlantic region of the United States. All junior and senior students, classroom faculty, and clinical preceptors (instructors) from one Commission on Accreditation of Athletic Training Education–accredited program were recruited for participation. Three groups, nonrandomly assigned as junior students (n = 9), senior students (n = 10), and instructors (n = 9), served as participants.
Due to the small number of participants, a post hoc analysis was performed for this exploratory study to determine the likelihood that true κ values would be calculated through the statistical analyses. The targeted strength of agreement was moderate, represented by a κ value range16 of 0.41 to 0.60; α value was .05; and minimum power value was 80%. Four separate post hoc analyses for the clinical skills and professional behaviors accuracy and reliability measures were conducted. Resulting κ values ranged from 0.42 for the professional behaviors accuracy measure to 0.78 for the clinical skills reliability measure. The post hoc analysis for the professional behaviors accuracy measure resulted in an 80% chance the true κ value fell within the substantial agreement range, and the remaining power analyses established an 80% chance the true κ values fell within the targeted moderate agreement range of 0.41 to 0.60.
Seventeen of the participants identified themselves as men and 11, as women. The instructors averaged 16.4 years (range, 5.0 to 41.0 years) of clinical athletic training experience and 7.7 years (range, 2.0 to 20.0 years) of experience evaluating athletic training students. The senior students had completed academic coursework related to this study 20 months before data collection and had finished 5 clinical rotations (3 full academic semesters, two 4-week summer sessions) at the time of data collection. Junior students had completed academic coursework related to this study 8 months before data collection and had finished 2 clinical rotations (1 full academic semester, one 4-week summer session) at the time of data collection.
Procedures
The data-collection instrument was adapted, with permission, from a textbook designed for athletic training clinical-skills documentation17 and was field-tested before use in this study. The data-collection instrument contained 5 clinical skills and 7 professional behaviors. The instrument used a 2-point nominal (yes/no) scale for assessment of 5 clinical skills (Biceps Femoris Manual Muscle Test [BFMMT], Lachman Test, Kleiger Test, Noble's Compression Test, Thompson Test). Each clinical skill included 3 subscales (patient position, clinician position, test performance). Participants based clinical skills scores on observed completion of the individual tasks needed for each clinical skill, as described in the data-collection instrument. The Appendix displays the individual items scored during data collection.
Participants assessed professional behaviors at the conclusion of the performance of the 5 clinical skills. Seven professional behaviors were scored using a 5-point Likert-type scale (5 = always, 4 = frequently, 3 = occasionally, 2 = rarely, 1 = never) to measure the frequency with which each behavior was observed during the clinical-skills demonstration. The 7 measured professional behaviors incorporated (1) skills performance, (2) confidence, (3) clear instructions, (4) respect for the patient, (5) patience, (6) demeanor, and (7) physical and emotional safety of the patient.
Each person participated in 1 treatment session conducted in a medical exam office that contained a standard treatment table in view of 3 chairs. All participants read and signed an institutional review board–approved informed consent document before the study began. For each treatment, a triad consisting of 1 participant from the instructor group and 2 participants from one of the student groups (junior or senior) were randomly assigned from the pool of available participants. This triad observed a participant from the assigned student group perform the 5 clinical skills. For example, if junior students were the assigned student group for a triad, 2 junior students and 1 instructor watched another junior student perform the clinical skills on a model patient during the treatment. The same occurred for senior students when they were the assigned student group for a treatment. The student performing the skills was instructed to interact with the model patient as he or she would in a real-life evaluation. Model patients had no previous experience as athletic training students or with the use of the clinical skills in this study. The instructions were read to the participants in each treatment session before demonstration of each clinical skill and before the scoring of professional behaviors.
Statistical Analyses
All analyses were performed with the use of AgreeStat 2013.1 software.18 Data are presented as percentage agreement and Cohen κ coefficient value (P < .05). The clinical-skills data were analyzed via the Cohen κ coefficient as a measure of interrater agreement of dichotomously scored data. To measure the accuracy of each student group in scoring clinical skills, the data from each student in a treatment session were compared with the instructor's data. This was repeated for every treatment session specific to the student group. The aggregate data from each student group across all treatment sessions were then used to calculate the κ value for that student group. To measure reliability of each student group in scoring clinical skills, the data from the 2 students in each treatment session were compared. Data were then analyzed in the same manner as for the accuracy measures for each student group. Kappa values were calculated for each of the 5 clinical skills and 3 subscales for each student group for both accuracy and reliability.
A linear weighted Cohen κ coefficient (κω) was used to analyze the accuracy and reliability of the professional behaviors to measure interrater agreement of the ordinal data. The data from the individual triads were collected and analyzed in the same manner for the professional behaviors as for the clinical skills. Kappa values were calculated as 1 value for all 7 professional behaviors for each student group for both accuracy and reliability.
RESULTS
Senior students were accurate for all clinical skills (range, 70.0%–89.7%, κ = 0.3735–0.5540, P < .001–.037), subscales (range, 71.1%–97.8%, κ = 0.2577–0.7887, P < .001–.019), and professional behaviors (48.6%, κω = 0.2559, P = .008). Senior students were reliable for the BFMMT (76%, κ = 0.4286, P = .003), Thompson Test (93.3%, κ = 0.6296, P = .014), test performance subscale (93.3%, κ = 0.3407, P = .002), and professional behaviors (54.5%, κω = 0.4094, P < .001). All results for senior students are presented in Table 1.
Junior students were accurate for the BFMMT (82.8%, κ = 0.5589, P < .001), Thompson Test (92.9%, κ = 0.7296, P < .001), Kleiger Test (71.4%, κ = 0.2593, P = .041), Noble's Compression Test (83.9%, κ = −0.0862, P = .005), patient position subscale (96.8%, κ = 0.7835, P < .001), and test performance subscale (77.1%, κ = 0.3236, P < .001). Junior students were reliable for the BFMMT (77.5%, κ = 0.3793, P = .030), Noble's Compression Test (90.6%, κ = 0.5200, P = .039), patient position subscale (92.6%, κ = 0.6250, P = .016), and test performance subscale (78.8%, κ = 0.3295, P = .023). Junior students were neither accurate nor reliable in scoring professional behaviors. All results for junior students are presented in Table 2.
DISCUSSION
Previously published studies in athletic training education found entry-level graduate students to be accurate peer assessors of videotaped recordings of clinical skill performance.5,6 The current study enrolled undergraduate students and incorporated a live skills demonstration, concurrent data collection among participants, and assessment of professional behaviors.
Senior students were accurate for all clinical skills, subscales, and professional behaviors. The junior students were accurate for the majority of the clinical skills (4 of 5) and subscales (2 of 3) but not the professional behaviors. The high number of incidences of significant agreement between students and instructors on clinical skills is on par with previous peer-assessment studies in athletic training education. Marty et al5,6 found students to be accurate graders compared with certified athletic trainers. The authors used percentage agreement to determine level of accuracy between students and certified athletic trainers in their grading of clinical skills recorded on video and reported very high levels of agreement for all skills (>94%).
Live-skills peer-assessment studies in medical and other allied health education reported excellent agreement between dental students and instructors on third-molar removal,19 very high levels of agreement between students and instructor grading on cardiopulmonary resuscitation skills,20 and moderate to substantial agreement between instructors and students during an objective structured clinical examination.12
The 2 groups in the current study had an equal number of statistically significant results for reliability (4). However, the seniors were reliable for professional behaviors, whereas the juniors were not. There are no published studies in athletic training education that used one-time interrater reliability measures to determine the reliability of students to assess their peers. Marty et al5 determined that graduate-level athletic training students were not reliable graders of their peers when the measurement was taken on one occasion, but they became more reliable as the number of measurements increased over time. In medical education, students improved in reliability over 7 semesters in assessing their peers.14 Self- and peer-assessment grades were significantly different from instructor grades but were significantly the same as each other.
Although medical education has produced studies involving peer assessment, there are no published studies in athletic training education investigating peer assessment of professional behaviors among athletic training students. During clinical-skills demonstration, student and faculty scores were compared on different aspects of professionalism with conflicting results. Despite a weak but significant correlation, medical students were significantly more lenient than instructors on grading communication skills,2 but students were found to be harsher graders than instructors for professionalism measures during a clerkship.13
Results of the current study suggest year in school may affect a student's ability to assess his or her peers. Senior students evaluated their peers at a similar level as instructors for all measured variables. However, junior students were neither accurate nor reliable for the professional behavior measures. The senior students had completed 1 more year of didactic and clinical education than the junior students at the time of data collection. With an extra year of clinical and classroom experience, the senior students were better able to consistently grade their peers in a similar manner to instructors. However, this finding is in contrast to previous studies performed by Marty et al.5,6 The authors in both studies determined year in school had no effect on the accuracy of athletic training students to assess their peers or provide accurate feedback to their peers.
The current study differs from previous athletic training peer-assessment studies5,6 in 2 key ways that may explain this difference in findings. First, the current study enrolled undergraduate students as participants, whereas the Marty et al5,6 studies used graduate-level students. Second, the participants in the Marty et al5,6 studies all had previous formal experience and training in use of peer-assessment skills, but the participants in the current study did not have prior formal experience or training in peer assessment. Further research is needed in order to determine whether academic year, academic level (graduate or undergraduate), and prior experience affect the ability of athletic training students to accurately assess their classmates on clinical skills.
Limitations and Future Directions
Limitations for this study include the low number of participants, lack of pilot study for data-collection instrument testing, and lack of training in peer-assessment practices among the participants. Participants were recruited from only 1 athletic training education program. This ensured all clinical skills used in the study were taught and assessed in a similar manner across the student participants. Although the small number of participants limits the generalizability of this study's findings to a larger population of athletic training students, the post hoc power analyses resulted in a favorable likelihood of reaching the target κ value range for all statistical analyses.
The limited pool from which to draw participants also affected the decision not to conduct a pilot study. Because the participant pool for the larger study was already small (28 total enrolled participants), the use of 1 or 2 participants from each group for a pilot study would not have provided enough data to draw conclusions about the data-collection instrument's value and would have decreased the number of participants available for the full study.
Multiple authors have stressed the importance of training participants before implementing a peer-assessment program.4,11,21–23 Although the current study was not a long-term peer-assessment program, it may have been beneficial for the participants to have a practice session with the data-collection instrument. All participants were able to look over the data-collection instrument and ask questions about it during the informed consent process and before the start of their data-collection session, but they did not use the instrument until data collection.
Future research into peer assessment in athletic training education should continue to use live skills evaluation, whenever possible. Compared with videotaped skills demonstration, assessment during a live skills performance more closely mimics the real-world conditions in which athletic trainers are evaluated during their careers by colleagues, patients, supervisors, and the public. In addition, investigating peer-assessment trends over multiple semesters in order to better understand whether year in school affects a student's ability to evaluate his or her peers is recommended. Finally, use of training and practice in peer-assessment skills is highly recommended for better research outcomes.4,12
CONCLUSIONS
Senior athletic training students assess their peers more accurately on clinical skills and professional behaviors than their junior student counterparts. Neither group of students demonstrated a significantly better ability to evaluate within their group during the study. Athletic training programs are encouraged to incorporate peer-assessment practices into their curriculums to better prepare students for careers in health care that use peer assessment on a daily basis.
Contributor Notes