Comparing Human- and ChatGPT-Generated Multiple-Choice Questions in Athletic Training Education

Instrumentation

We constructed a survey for athletic training faculty with 3 sections. The first section included consent and 2 questions to determine inclusion. Participants were asked if they had ever taught in a didactic setting and if they were familiar with the format of BOC-style exam questions. Participants were included only if they answered yes to both questions. The next section asked participants to evaluate 10 pairs of MCQs. Each pair included a human-generated question and a ChatGPT-generated question on a similar topic. Please see Table 1 for an example of 2-question pairs.

Table 1.Example of Question Pairs (Domain 2: Assessment, Evaluation, and Diagnosis)

View Fullscreen

Ten human-generated MCQs were selected by the research team from previously used program exams. The research team used Microsoft Word’s grammar and spell check functions when the questions were originally created. Two questions were chosen from each of the 5 athletic training domains. Each question met the following criteria: (1) multiple-choice format with 5 answer options, (2) Bloom taxonomy application level or higher, (3) corrected item-total correlation coefficient of 0.3 or above on a recent exam, and (4) aligned with BOC exam question creation guidelines.

ChatGPT version 3.5 (current free version in February 2024) was used to create 10 MCQs with similar content to human-generated questions. To align ChatGPT-generated questions with human-generated questions, the prompts inputted into ChatGPT used a similar template to that of Cox et al, which included a specific topic and Bloom taxonomy level.¹⁸ For example, “Create an athletic training BOC-style, multiple-choice question about [topic] at the application level of Bloom’s taxonomy with a short or medium prompt. Create 5 answer options.” For example, one ChatGPT prompt stated: “Create an athletic training BOC-style, multiple-choice question about evaluation of tarsal tunnel syndrome at the application level of Bloom’s taxonomy with a short or medium prompt. Create 5 answer options.” See Table 1 for the resulting MCQ.

Participants were not aware of the origin of the question (human or ChatGPT). Faculty participants evaluated each question on grammar, clarity, difficulty, terminology, and suitability using a 5-point Likert scale (1 = very poor to 5 = very good). Participants responded to 2 separate questions about clarity, 1 for the stem, and 1 for the answer options. They also evaluated whether each question would be at an appropriate level of difficulty for an entry-level athletic trainer. Participants rated the medical terms and abbreviations for accuracy/appropriateness and the suitability of each question to effectively address entry-level athletic training content. Participants were then asked which question from the pair they would be more likely to use in an examination (question 1, question 2, neither). The final section included demographic questions. The survey was pilot tested by 2 athletic training faculty members not affiliated with this study. The survey was revised to improve clarity and decrease completion time.

A student quiz was also developed to further evaluate question quality/difficulty. Each MCQ used in the faculty survey was uploaded to Canvas, a web-based learning management system. Current second-year master’s students nearing graduation were asked to complete the 20-question quiz. Scores from this 20-question quiz were used to determine the corrected item-total correlation coefficient and item difficulty scores for the ChatGPT-generated and human-generated MCQs used in the survey sent to athletic training faculty.

Participants

Masters-level professional athletic training programs were identified in each state using the Commission on Accreditation of Athletic Training Education website. Faculty contact information was collected from publicly available directories on the selected institutions’ website. Second-year students enrolled in host institutions’ master’s-level professional athletic training program were recruited. All student participants are known to the lead investigator.

Procedures

The research team emailed 653 athletic training faculty members teaching at master’s-level professional athletic training programs to request their participation in this study. We sent a reminder email 3 weeks later. The survey was hosted on the Qualtrics platform (Qualtrics). After providing informed consent, faculty members were asked to complete the survey. Completion of all survey items took approximately 20 minutes. Data were collected anonymously.

We sent an email to 13 current second-year master’s-level athletic training students requesting they complete a 20-question, multiple choice quiz. This quiz did not affect student participants’ grade in any course. Confidentiality was assured, and informed consent was obtained from each participant.

Data Analysis

Faculty survey results were exported from Qualtrics into SPSS version 26 (IBM Corp). Questions related to quality, clarity, relevance, and difficulty were rated on a 5-point Likert scale (1 = very poor to 5 = very good) and compared using the Wilcoxon signed rank test. Questions about MCQ preference were analyzed descriptively as frequency and percentage.

For the 20-question student quiz, we used the Canvas quiz and item analysis report that provides the corrected item-total correlation coefficient and item difficulty score for each question. The corrected item-total correlation coefficient for each question was examined to determine if a question had good distractions and provided good discrimination (score of 0.25 or above) or if a question may have been miskeyed or confusing for students (negative score). Item difficulty scores were used to identify questions that may have been too easy or too difficult.

Grammar

Faculty participants rated grammar as acceptable to good (range, 3.4–4.3) for all questions. Human-generated questions ranged from 3.4 to 4.0, whereas ChatGPT-generated questions were rated from 3.6 to 4.3. Table 3 shows significant differences in grammar quality for 4 question pairs. In these 4 question pairs, faculty participants rated the ChatGPT-generated questions higher, indicating better grammar.

Table 3.Question Item Ratings^a

View Fullscreen

Stem

Faculty rated the quality of the question stem as acceptable to good, ranging from 3.2 to 4.2. Human-generated questions ranged from 3.2 to 4.1, whereas ChatGPT-generated questions ranged from 3.2 to 4.2. Five question pairs exhibited statistically significant differences in the quality of the question stem (see Table 3). In 3 cases, faculty participants rated the ChatGPT-generated questions higher, indicating a better stem.

Answers

Faculty rated the quality of answers for both human- and ChatGPT-generated questions as acceptable to good, ranging from 3.2 to 4.3. Seven question pairs showed statistically significant differences in the quality of answers (see Table 3). In 4 of these pairs, human-generated questions were scored higher.

Difficulty

Athletic training faculty rated the difficulty of the questions as acceptable for entry-level athletic training students, ranging from 3.0 to 3.9. Human-generated questions ranged from 3.0 to 3.9, whereas ChatGPT-generated questions ranged from 3.1 to 3.9. Statistically significant differences in perceived difficulty ratings were observed in 5 question pairs; in 3 of these pairs, ChatGPT-generated questions were perceived as having a better or more appropriate level of difficulty for entry-level athletic training students than their human-generated counterparts (see Table 3).

Terms

Athletic training faculty rated the use of terms as poor to good, ranging from 2.9 to 4.1. Human-generated questions ranged from 2.9 to 3.9, whereas ChatGPT-generated questions ranged from 3.4 to 4.1. Four question pairs showed statistical differences. In 3 of the 4 question pairs, ChatGPT-generated questions were rated higher, indicating more appropriate terminology use (see Table 3).

Suitability

Athletic training faculty rated the suitability of questions as poor to good, ranging from 2.8 to 4.0. Human-generated questions ranged from 2.8 to 3.8, whereas ChatGPT-generated questions ranged from 2.9 to 4.0. In 7 question pairs, significant differences were observed, with human-generated questions deemed more suitable in 4 of the question pairs (see Table 3).

Question Preference

There was no clear preference for either type of MCQ. When comparing the question pairs, participants preferred 5 questions generated by ChatGPT and 4 questions generated by humans. In 1 instance, preferences were evenly split between the 2 types of questions, as shown in Table 4. Each of the 5 athletic training domains included 2 question pairs. In 3 of these domains, participants favored 1 ChatGPT-generated question and 1 human-generated question. Overall, question preference was balanced across the different athletic training domains.

Table 4.Question Item Preference

View Fullscreen

Student Results

Eleven student participants completed the 20-question student quiz, with a response rate of 85%. The majority of student participants were under the age of 25 (82%). Table 5 summarizes student participant demographics.

Table 5.Student Participant Demographics

View Fullscreen

Three human-generated questions and 1 ChatGPT question had an item difficulty score above 0.85, indicating that they may have been too easy (see Table 6). Only 1 human-generated question and 1 ChatGPT question had an item difficulty score below 0.30, indicating it may have been difficult. The remaining questions fell within an acceptable range. Six human-generated questions and 5 ChatGPT-generated questions achieved a corrected item-total correlation coefficient value of 0.25 or above, indicating that these questions had good distractors.¹² There were 2 human-generated and 5 ChatGPT-generated questions with a value below 0.25 and 3 ChatGPT-generated questions that received negative values, which may indicate that a question is miskeyed or confusing. All participants answered 2 human-generated questions correctly. In these cases, a corrected item-total correlation coefficient score could not be calculated and “NA” (not applicable) appears in the table.

Table 6.Student Results

View Fullscreen

Overall, human-generated and ChatGPT-generated questions showed a range of difficulty (item difficulty values between 0.27 and 1.0). Higher scores (eg, 0.80) mean that more students answered the question correctly and the question was easier. Lower scores (eg, 0.20) mean that fewer students answered the question correctly and the question was more difficult. There were 4 human-generated and 3 ChatGPT-generated MCQs with item difficulty scores over 0.80. There were no questions with an item difficulty score below 0.27.

Grammar and Terminology

Athletic training faculty rated grammar as acceptable to good for both human-generated and ChatGPT-generated questions. The ChatGPT-generated questions had slightly higher grammar scores in 4 cases, indicating that faculty might want to consider using ChatGPT to improve grammar in their human-generated questions. Additionally, for most of the MCQs, the use of medical terms and abbreviations was considered accurate and appropriate. These results align with previous studies showing that ChatGPT can create MCQs that follow general rules of grammar and syntax.^18,19 For example, Cox et al found that nursing faculty rated ChatGPT-generated questions and nursing faculty–generated questions similarly in their clarity and grammar.¹⁸ Nasution found that 73% of biology students thought the AI-generated questions in their study were without grammatical or conceptual errors.¹⁹ The author noted that although they did encounter a language or sentence issue in a question created by AI, an expert could fix the mistake during the review process.¹⁹ Therefore, ChatGPT appears to be an effective tool to generate MCQs that closely reflect natural language and use acceptable medical terminology, but careful review is still needed.

Question Stems and Answer Options

Athletic training faculty also found the question stems and answer options acceptable for both human-generated and ChatGPT-generated questions. Although faculty noted a wider range in the quality of ChatGPT-generated question stems, the ChatGPT questions scored higher in 3 of the 5 cases of statistical difference. Previous studies examining ChatGPT-generated MCQ quality have found mixed results.^3,18,19,24 For example, Cox et al found that ChatGPT generated clear stems with correct answers and appropriate distractors for nursing content, although the authors noted that questions could be improved by faculty with content knowledge.¹⁸ Cheung et al found similar results for medical education MCQs.¹⁷ However, a range of content accuracy rates was found in a review of 23 studies that used ChatGPT to create medical education MCQs.²⁵ Additionally, Ngo et al found incorrect answers and explanations in most immunology MCQs generated by ChatGPT and remarked that 43% of questions would need significant changes before they could be used.³ Our results are consistent with Ngo et al in that our human-generated questions scored better for correct answers more often (57%; 4 of 7) than the AI-generated questions when question pairs were statistically different from each other.³ Overall, ChatGPT provides a promising tool for automatically generating MCQs, but faculty should expect to review questions and provide necessary changes to ensure accuracy and clarity for both questions’ stems and answers.

Question Difficulty

Athletic training faculty rated the difficulty of each pair of MCQs as acceptable. Most question pairs were perceived as having an equally appropriate level of difficulty, whereas 3 ChatGPT-generated questions and 2 human-generated questions were rated as having a more appropriate level of difficulty than their counterpart question. Overall, both types of questions could effectively address entry-level athletic training content.

Results from the student quiz data showed that most human-generated and ChatGPT-generated MCQs were within an acceptable level of difficulty. There were 4 questions that could have been considered too easy and only 2 questions that might have been too challenging. The faculty difficulty Likert score ratings did not forecast student performance for either the human-generated or ChatGPT-generated questions (Tables 3 and 6). This might be accounted for by the difference in measurement scales. Faculty were asked if the question had an appropriate amount of difficulty rather than if it was a more difficult question.

Item Discrimination

Results from the student data showed that most human-generated MCQs had good item discrimination, with only 2 questions showing a need for revision. However, half of the ChatGPT-generated questions had values indicating poor discrimination. Additionally, 3 of these questions had negative values, indicating that these questions may have been ambiguous or confusing. This underscores the importance of expert review prior to using new questions on exams and the need for analyzing the data generated when these questions are used in course assessments.

Suitability and Preference

Suitability ratings were mixed, ranging from poor to good for both human- and AI-generated questions. There was no clear preference overall, with participants favoring an almost equal number of human- and ChatGPT-generated questions when there was a statistical difference in suitability. The athletic training domain did not seem to impact human versus ChatGPT question preference. It was common for at least 1 human-generated question to be preferred per domain. In 7 of the 10 question pairs, there was a statistically significant difference in suitability. In each of these 7 pairs, the question considered “more suitable” was also the preferred question. This trend was also seen with difficulty ratings and question preference, where statistically significant differences were found in difficulty ratings in 5 question pairs, and the preferred question was also a question that was considered to have a more appropriate level of difficulty.

Recommendations for Athletic Training Faculty

Given the results of this study and the increased demands on faculty members, we recommend that athletic training educators consider using ChatGPT to generate quality quiz and exam questions. For faculty who have not used ChatGPT, they could consider using our simple template to generate an initial draft of a question, then using an approach similar to that proposed by Zuckerman et al, in which a human instructor reviews and edits the question to remove distractors that were not taught, changes item wording to match what students had learned, and adds clinically relevant details to the question stem.²² With this approach, educators can leverage ChatGPT to reduce the burden of crafting effective questions, while still ensuring that questions are tailored to the content and level of the students.

Once a reasonable question has been produced, we can use multiple methods to ensure that a question is fair and valid. First, we can identify and correct common writing flaws such as excess verbiage in the stem, use of implausible distractors and absolute terms (eg, always, never), and making the correct answer more detailed or longer.^27,28 After a test has been administered, we can use item analyses to identify questions that need to be edited or removed.¹² If an exam is housed in a university’s learning management system such as Canvas, item analysis results are provided for instructors. In this study, obtaining results for item difficulty and corrected item-total correlation coefficients identified multiple questions that could be improved with modifications.