The Effects of an Electronic Audience Response System on Athletic Training Student Knowledge and Interactivity

Study Design

We used preintervention and postintervention evaluations of students' knowledge and degree of interactivity via a researcher-developed survey. Athletic training students were enrolled in an introductory athletic training course at a Commission on Accreditation of Athletic Training Education (CAATE)–accredited public institution during the time of participation. This introductory athletic training course was selected for the following reasons: (1) it is an introductory course that does not require previous knowledge of athletic training; (2) it has 2 sections offered by the same co-instructors; (3) the different sections are offered on the same days of the week and both are morning classes; (4) the course is a very typical athletic training course that teaches both theory and application of the theory and skills learned in the course; and (5) the course has a natural break in course objectives after unit 1. The theory portion was taught in a tradition lecture/discussion format. The skill application was taught via a laboratory format. The course participants met for 2 hours twice a week. The first hour of each day was theory and the second hour was lab based. The ARS was only used during the theory portion of the course. The main objectives of the first unit of this course related to basic emergency-related athletic training skills (eg, vital assessments and cardipulmonary resuscitation [CPR]). University institutional review board approval was obtained before data collection began.

Participants

Undergraduate athletic training students registered in an introductory athletic training course were asked to participate in this study. All participants (N = 69; control n = 35; experimental n = 34) had declared athletic training as their major but had not yet been admitted into the athletic training program. Participants were allowed to select which section they enrolled in based on their personal schedules. Each participant was asked to complete a self-reported demographic survey at the beginning of the study. Based on self-report questionnaire responses the control group was composed of 21 females and 14 males. The experimental group was made up of 19 females and 15 males. The Table provides additional selected demographic information on the participants in the 2 groups. Additional demographic information revealed the 2 groups combined consisted of 57 (82.6%) Caucasian, 3 (4.3%) Asian American, 4 (5.8%) African American, 2 (2.9%) Latino, 2 (2.9%) Native American, and 1 (1.4%) Pacific Islander students. Within this sample, 44 (63.8%) were freshman, 15 (21.7%) were sophomores, and 10 (14.5%) were juniors.

Table.  Participant Demographic Information Within Each Group

View Fullscreen

Instrument

We developed and used the Knowledge and Interactivity Survey (KIS) to assess basic athletic training knowledge and degree of individual interactivity for this study. The KIS consisted of 10 questions (multiple choice and fill in the blank) to assess knowledge learned within the course and 10 statements examining an individual's perception of his/her degree of interactivity with the course. The 10 knowledge-based questions were generated from the content of the required textbook for the course. Knowledge questions in the KIS instrument were examined for face validity by a panel of experts (n = 8). No modifications were deemed necessary based upon feedback. Reliability of knowledge questions in the KIS instrument was determined with a pre-post design, and independent sample t tests in a sample of undergraduate athletic training students (n = 20) who had previously completed the course used in the study. We identified no significant differences (t₁₈ = −1.372, P = .187) presurvey and postsurvey. Additionally, a paired samples correlation revealed high intercorrelations between presurvey and postsurvey (r = 0.93), thus making this a reliable instrument.

The interactivity questions were adopted from a previous study²⁶ conducted to establish reliability and were validated by a panel of experts (n = 8). Cronbach α was calculated for all of the items in this pilot study at the value of 0.86, which suggests that the instrument is highly reliable. Interactivity was assessed using a Likert scale measuring responses to 10 questions ranking the degree to which the participant felt he interacted in the class. The scales included 9 ordered choices to posed questions ranging from 1, which indicated strongly disagree, to 9, which indicated strongly agree. The KIS instrument in its entirety may be found in the Figure.

View Figure

• Download as Powerpoint
• Download Figure

Procedures

For this study, we selected one section of the introductory athletic training course to serve as the control group and the second section to serve as the experimental group. The only modification to the experimental group was the inclusion of the ARS technology (Turning Technologies, Youngstown, OH) during the lecture/discussion portion of the course.

On the first day of class, all participants were informed of the study, given a chance to ask questions, and asked to complete a consent form if willing to participate. Each participant was given a 2-digit number to place on the presurvey and postsurvey KIS instrument to track responses. Participants in both the control and experimental groups filled out the KIS instrument at the end of the first week of the course (approximately 4 hours into the course). This provided the baseline (pre-) measurement for all participants' knowledge of concepts related to the specific course objectives in the first half (unit 1) of the course and each individual's self-assessed degree of interactivity related to the course.

PowerPoint presentations were used in both sections of the course to deliver course content to participants and were identical in content. However, each PowerPoint presentation for the experimental group contained 5 to 8 additional slides with ARS questions presented at a pace of approximately 1 question every 20 minutes. Pacing and placement of questions using the ARS were critically considered. Previous researchers^15,27 have suggested that ARSs may negatively affect cognitive gains and a student's interactivity in class if questions are posed too frequently or are not presented at an appropriate cognitive level.

At the conclusion of the study in unit 1 (8 weeks into the 16-week course), both sections were given a paper copy of the KIS instrument to complete during class. Participants were informed this was the posttest for the study. Upon completion participants were thanked for their involvement in the study. Results were analyzed between the control and experimental sections as well as within the 2 sections for patterns.

Data Analysis

The data were imported from the presurvey and postsurvey KIS instruments into IBM SPSS Statistical Package for Windows (Version 20) for statistical analysis (SPSS Inc, Chicago, IL). Following the advice of Tabachnick and Fidell,²⁸ we screened the data for accuracy, missing data, normality, and outliers. No issues were identified with inaccurate data entry or recording, nonnormality, or outliers. One participant from the control group was removed from the data set as a result of excessive missing data. To quantify the effects of the ARS, the difference between responses to the presurvey and postsurvey KIS instrument was calculated and grouped responses into 2 sets of pre/post pairs—knowledge and interactivity—based upon the variable addressed in each survey question. Survey questions 1 through 10 addressed knowledge, while survey questions 11 through 20 addressed interactivity. Paired samples t tests examined the effect of the ARS on the variables. Repeated-measures analysis of variance (ANOVA) tests examined the differences between the control group and the experimental group in terms of the knowledge and interactivity variables.

The dependent variables (knowledge and interactivity) were examined with a Pearson product-moment correlation coefficient²⁹ to determine whether they were sufficiently correlated to conduct a multivariate ANOVA. The dependent variables were weakly correlated at r = 0.340, P = .02, so the decision was made to conduct all further analyses with mixed-measures ANOVAs.²⁸

Effect on Knowledge

A mixed-measures ANOVA and Box Test of Equality of Covariance Matrices found no significant difference between experimental and control groups, F_(3,928881) = .332, P = .803, ns, indicating that the assumption of homogeneity of covariant matrices was not violated. An interaction was discovered for the effect of instruction and use of the ARS, F_1,59 = 5.89, P = .018, η²_p = .091, indicating that the acquisition of knowledge in the experimental group (7.97 ± 1.49) was greater than for the control group (7.24 ± 1.75). This finding supports using ARS within an introductory athletic training course to increase student knowledge acquisition.

Effect on Interactivity

A mixed-measures ANOVA and with Box Test of Equality of Covariance Matrices found a significant difference between experimental and control groups, F_(3,928881) = 5.65, P = .001, indicating that the assumption of homogeneity of covariant matrices had been violated. Because the covariance matrices of the dependent variables could not be assumed to be equal across groups, the interpretation of the ANOVA was conducted using a Greenhouse-Geisser correction.

There was a main effect for interactivity, F_1,59 = 5.40, P = .024, η²_p = .084, indicating that individual interactivity in the course increased among participants from presurvey (7.16 ± 1.23) to post (7.56 ± 1.08); however, there was no overall interaction between interactivity and group membership.

Effects on Knowledge

The participants demonstrated a statistically significant increase in knowledge acquisition within an introductory athletic training course covering assessment of vital signs and recognition and management of cardiac and breathing emergencies with the use of an ARS. Furthermore, a difference between the control and experimental group was demonstrated when examining acquisition of knowledge. When using an ARS within a section of an introductory athletic training course, participants demonstrated higher knowledge acquisition than did the control group. The interaction suggests that the use of ARSs makes a small, but statistically significant, contribution to knowledge acquisition among participants who use them when compared with a course that does not use ARSs. Of the previous authors^1,3,12,16,22 who examined knowledge acquisition through the use of ARSs during lectures, positive findings were also reported. One researcher³⁰ incorporated ARS into an undergraduate emergency health course over a 4-week unit of the semester. This course was interprofessional in design, as it included students from paramedics/nursing, occupational therapy, physiotherapy, and health sciences. Among this group, 77% reported an increase in knowledge acquisition in the ARS unit.³⁰ This is supported by our findings.

Instructors' evaluations of ARSs as a formative learning assessment tool reveal that the devices are an effective way to discover which material students do and do not understand.³¹ Therefore, when a student responds to a multiple-choice or true-false question presented with an ARS during a lecture, the instructor can efficiently evaluate the materials presented. This enables the instructor to make immediate modifications to lessons to address student challenges. Additionally, ARSs benefit students by providing a method by which to gauge understanding of course material. Compared with hearing their classmates' verbal responses to questions posted in lectures, answering questions through an ARS more accurately allows a student to assess her level of self-mastery of material.³² Identification of deficiencies can provide a student direction in what material she needs to spend greater time reviewing after class.

The uses of an ARS as they relate to teaching, learning, and assessment are widespread and applicable in a variety of situations. One factor for faculty to consider is whether the ARS questions are graded or simply formative with no grades attached. In the present study, we used a formative ARS study design, in which participants and the instructor were able to evaluate the student's comprehension from assigned reading and presented materials. All participants, both in the control and experimental groups, were assigned readings before each of the lessons. However, the completion of these readings along with the participants' achieved level of understanding of the materials were not evaluated. It could be hypothesized that if the participants knew they would be graded on ARS questions, they may have spent more time on reading and understanding content before class. Some researchers³³ suggested that graded ARS questions insure class attention and effort in preparation. However, other researchers³⁴ evaluating the impact of ARSs found that students rated the system less positively when it was used for a graded rather than a formative purpose. Additionally, Cain and Robinson¹¹ concluded that to ensure the genuineness of the feedback process, formative questions should not be graded. Further research in athletic training education may warrant an exploration into knowledge acquisition over time with graded ARS exercises.

Effects on Interactivity

Both the control and experimental groups indicated statistically significant increases in classroom interactivity. However, both groups increased at the same rate, which did not support our hypothesis that differences would exist between the groups when using ARS technology. The benefits of an ARS on participant interactivity, emotion, and satisfaction in lecture-based courses have long been demonstrated^10,16,26 to have positive effects in a wide variety of educational disciplines. In one study,¹⁰ a large introductory psychology lecture class examined the impact of an ARS on student interactivity and satisfaction between 2 classes. One class was presented in a traditional lecture format with allowance for questions through informal hand raising while the other class used an ARS for formal review questions throughout the lecture. Using a Likert scale evaluation, students reported their impressions of the class, including their perception of the individual degree of interactivity in the class and with the instructor. Findings demonstrated significantly higher interactivity scores, greater positive emotion, and satisfaction with the course.¹⁰

We suspect that the structure of this athletic training course may have had an impact on interactivity measures within this study. Many athletic training courses are designed to integrate lecture material with hands-on skill practice. The objectives covered in this study included hands-on practice of CPR techniques, heart and lung auscultation, and vital sign assessment in a laboratory session after the associated lecture. Participation in these hands-on group interactions may have been the cause of an overall positive effect on the students' classroom interactivity in both the control and experimental groups, as demonstrated by the positive gains in both groups. Therefore, if there were any differences in interactivity between the groups related to the ARS in lectures, they may have been overshadowed by the group interaction in the laboratory component of the course that followed. Further studies in athletic training education may warrant exploration of ARS in terms of individual interactivity in the laboratory component of a course.

DeBourgh¹⁵ found the use of ARSs within a baccalaureate nursing education program effective in promoting students' acquisition and application of advanced reasoning skills in addition to increasing students' interactivity within the course. Audience response system questions were carefully designed to facilitate discussion within the course. Often questions were purposefully vague and designed to stimulate engaging debate. Beatty³⁵ described this questioning technique as successful in sensitizing students to the clinical integration of concepts. The goal of this type of questioning is not to promote memorization of factual knowledge but instead to demonstrate critical thinking strategies. Athletic trainers are among those health profession practitioners that require well-developed critical thinking abilities. However, as Knight³⁶ explains, information and new skills must be absorbed and practiced before they can be converted into performance knowledge. Educators can facilitate the acquisition of critical thinking skills through carefully designed methods such as the ARS questions DeBourgh¹⁵ described. Therefore, careful consideration should be given to the type of ARS questions (factual knowledge versus discussion promoting) in relation to the overall goals of the lesson. Related to athletic training education courses, in order to maximize the interactivity and effectiveness of the ARS, the instructor is challenged with assessing the level to which students have absorbed factual knowledge and with identifying appropriate opportunities to transition to critical thinking.