Development and Validation of the Athletic Training Student—Soft Skills Assessment Instrument
Education of athletic training students should include the acquisition of soft skills. Evaluation of soft skills is important but challenging. Multisource performance reviews can provide students with feedback from different viewpoints, which can be used to promote self-awareness and encourage reflection. To develop and validate a new multisource soft skills assessment tool for master’s level professional athletic training students. Cross-sectional study. Web-based questionnaires. Nineteen participants evaluated the instrument for face and content validity. These participants included 5 athletic training educators, 7 athletic training preceptors, and 7 students currently enrolled in a graduate-level, Commission on Accreditation of Athletic Training Education-accredited athletic training education program. Participants in the first pilot test included 22 students, 22 preceptors, and 3 instructors. Participants in the second pilot test included 28 students, 32 preceptors, and 3 instructors. A 3-stage process was used to develop and validate the instrument. The process included a literature review and formation of 65 survey items organized into 11 themes. Questions were reviewed by students, preceptors, and instructors. Responses were analyzed using content validity ratio to help identify items needing revision, deletion, or substitution. All open responses were considered. Based on results, 1 question was removed, and 2 questions were updated to improve clarity. The updated Athletic Training Student—Soft Skills Assessment instrument includes 64 questions. In the final stage, reliability analysis was conducted on 2 samples. Internal consistency was examined with Cronbach α and interitem correlations. Using a rigorous content validation approach provided evidence that the instrument was comprehensive, clear, and complete enough to establish the tool’s credibility in the preliminary stages. The outcome of this study is a multisource soft skills assessment tool that can provide students with formative feedback.Context
Objective
Design
Setting
Patients or Other Participants
Data Collection and Analysis
Results
Conclusion(s)
KEY POINTS
-
The design of the Athletic Training Student—Soft Skills Assessment instrument allows students to receive formative feedback from clinical and academic sources while comparing results with a self-assessment.
-
This study used a rigorous content validation approach to provide evidence of instrument reliability and validity.
-
Future researchers should examine student perceptions of the feedback and the ability of the instrument to assist educators in designing effective interventions to improve student soft skills.
INTRODUCTION
Job success in athletic training is closely tied to quality soft skills. In fact, soft skills are sometimes referred to as employability skills since they can influence a person’s ability to find work and to be productive and successful in a career.1–3 Other terms for soft skills include interpersonal skills, transferable skills, and essential skills because they include the ability to effectively deal with people, can apply to many different settings, and are considered basic abilities required to succeed professionally.4–7 Soft skills are character traits, attitudes, qualities, and behaviors.2 Time management, communication skills, adaptability, resiliency, confidence, ability to learn from mistakes, and ethical practice are considered soft skills that are noted as valuable to athletic trainers, employers, and preceptors.8–11 A connection also exists between soft skills and academic success.12–14 For example, superior academic performance in college and higher study satisfaction are associated soft skills such as responsibility, dependability, and self-discipline.14
Education of athletic training students should include the acquisition of knowledge, technical skills (hard skills), and soft skills. Assessment of knowledge and technical skills are a common part of education programs; however, evaluation of soft skills is done with less frequency.15 This may be due to the difficulty in measuring them objectively and consistently.16,17 Despite the challenges, measurement of soft skills is important. Without measurement, it may be difficult to identify areas of strength or weakness, to design appropriate interventions, and to know if those interventions are effective in aiding students’ soft skill development.
Currently, no gold standard for measuring soft skills exists. Methods for assessing the soft skills of students in health professional programs include but are not limited to simulations, patient assessment questionnaires, paper-based tests, and self-administered rating scales.18 Soft skills are frequently assessed through self-reporting questionnaires. However, gathering opinions from multiple sources may provide a more comprehensive assessment than can occur when only a single point of view is used.19 Multiple-source performance reviews are often used in the business community, where different categories of people (eg, peers and supervisors) contribute to the assessment of an individual. This type of feedback instrument collects input from those who interact most frequently with the subject of the feedback. Access to feedback from multiple sources and different viewpoints can help identify blind spots, promote self-awareness, and encourage reflection.20
If student perception of competence is not accurate, it may hinder growth and development. Researchers have shown that a gap can exist between the self-ratings and the ratings provided by employers.19 Identifying areas where students overappraise or underappraise their soft skills can promote self-awareness which, in turn, could contribute to meaningful change and professional growth.20 For example, Sadeghi and Loripoor21 used a 360° evaluation method to assess professional behavior and clinical skills of nursing students working in a pediatric ward. Students completed a self-evaluation and were also evaluated by their peers, an instructor, the clinical nurses in the ward, and the mothers of the pediatric patients. Authors advocated for the continued use of this type of assessment in nursing education, concluding that providing students with feedback from multiple perspectives could lead to improved patient care. Additionally, encouraging results have been seen when using multisource feedback methods in medical programs.20–25
Although soft skills are commonly appreciated among all health care professions, having an assessment tool designed specifically for the field of athletic training education may be valuable. A specific, multisource soft skill assessment instrument would be useful because the field encompasses both academic and clinical settings, where multiple perspectives are warranted for student learning. Currently, an assessment instrument designed to include soft skill ratings from athletic training students, preceptors, and instructors does not exist in the literature. The aim of this paper is to describe the development and validation of the Athletic Training Student—Soft Skills Assessment (ATS-SSA) instrument, a new multisource soft skills assessment tool for master’s level professional athletic training students. The objective of this study was to create an assessment tool that would (1) provide athletic training students with formative feedback to promote awareness, reflection, and growth and (2) provide athletic training program instructors with a tool to help identify areas of strength or weakness for the purpose of designing appropriate interventions to help students improve.
METHODS
The present study used methodological research. The design methodology used to develop and validate the ATS-SSA instrument follows multiple stages (Figure). This involved (1) defining and operationalizing the construct through reviewing the literature and consulting experts, (2) soliciting feedback and establishing internal validity, and (3) pilot testing the instrument to continue collecting data on validity.26–30 The design stages, activities, and outcomes are summarized in Table 1. The focus of this study is on the first 2 stages of instrument development but begins to address the third stage. The Institutional Review Board at the investigator’s institution reviewed and approved the methods, protocols, and instruments for each part of this study. The findings are reported using the recommendations for reporting the results of studies of instrument and scale development and testing.31
Citation: Athletic Training Education Journal 18, 3; 10.4085/1947-380X-22-027
Stage 1: Substantive Stage: Define Constructs
The objective of this stage was to define and measure the construct and assess content. Four guiding questions for this stage included: (1) How are soft skills defined in the literature? (2) Which soft skills are noted as being important in the literature? (3) How are soft skills measured in the literature? (4) What recommendations exist for defining and measuring soft skills? A thorough review of literature was conducted using the keywords such as soft skills, and employability skills. Articles with a focus on higher education, business, and health care were included. Particular attention was paid to which soft skills are considered important across health care disciplines and specifically within the profession of athletic training. An extensive literature search was also conducted to identify existing instruments with good psychometric properties that could be used as a frame of reference for the new instrument. From this research, a prototype survey was created. The survey was organized into 11 soft skill themes from previously published research.32 These themes include a range of soft skills reported in the literature, such as accountability, reliability, honesty, and a respectful demeanor, which are noted in the 2020 Standards for Accreditation of Professional Athletic Training Programs.33 Recently, Kutz and Stilner (2021)15 found that the soft skills contained in these themes are perceived to be very important to athletic training education program directors.
Each theme is operationally defined in the survey. Soft-skill constructs are composed of several related components. Thus, questions are included to assess different components within a soft-skill theme. Care was taken to ensure each question represented an observable behavior. Question format followed established guidelines.34 Questions are simple, short, and written in a language familiar to the target respondents. Each question only assesses a single component or behavior. Each soft-skill theme includes 3 to 10 questions. The entire survey included 65 questions deemed adequate to represent the constructs of interest.
The length on the survey was thoughtfully considered. To meet the goals of the instrument (ie, provide meaningful and actionable feedback), it was important to include the evaluation and ratings of key behaviors that contribute to a soft skill. Also, using a large pool of items in the early stages of questionnaire development was recommended to ensure the survey included items that adequately represent the construct and to minimize measurement error.35 However, lengthy surveys may promote fatigue and increase the likelihood of meaningless answers and poor data quality.36–38 To combat potential respondent fatigue, design choices were made based on recommendations in the literature and a survey construction expert. This included limiting each survey page to questions related to 1 theme at a time, avoiding the necessity to scroll when possible and keeping the layout simple and visually appealing to reduce cognitive burden.39,40 In the final steps of this stage, the survey was reviewed by a survey construction expert to ensure the instrument was grammatically correct and free of item construction flaws (vulnerabilities contributing to item misinterpretation). All recommended revisions were completed.
Stage 2: Structural Stage: Obtain Feedback to Establish Content and Face Validity
The objectives in this stage included (1) soliciting feedback, (2) revising the instrument categories and descriptions as needed, (3) establishing face and content validity, and (4) determining if the soft-skill constructs match how the potential respondents think about them.27 Two research questions were associated with this stage: (1) Does the instrument measure what it intends to measure? (2) Does the instrument accurately represent the behaviors associated with soft skills?
Three versions of the ATS-SSA instrument were created in Qualtrics. The 3 surveys included (1) an instructor version, (2) a preceptor version, and (3) a student version. The questions were very nearly identical with only slight wording changes when necessary to make the instrument appropriate for the intended audience. Surveys were sent to a convenience sample of 63 individuals (21 educators, 20 preceptors, and 22 students). Educators and preceptors from different parts of the country working in different settings were chosen as participants. Individuals were recruited if they currently worked for or with a Commission on Accreditation of Athletic Training Education (CAATE)-accredited athletic training program. Educators employed in different roles (ie, program directors, clinical education coordinators, and faculty) and preceptors with various years of experience were invited to participate. Some preceptors affiliated with the author’s home institution were included in the recruitment pool. Students with prior knowledge of and experience with the soft-skill themes used in the design of this instrument were also recruited. These students were currently enrolled in a CAATE-accredited graduate athletic training program and were from the author’s home institution. Each recruited student had previously completed a 5-week course that included instruction and activities related to the themes included in the survey. Additionally, each student had completed at least 14 weeks of supervised clinical experience. All potential participants were contacted via e-mail and asked to assess each item in the instrument for relevance and clarity. Participants rated items using the following scale (1 = item is clear and relevant, 2 = item is clear but not relevant, and 3 = I do not understand this item, or it is unclear). For each theme, participants were asked if any behaviors were missing. Responses were open ended. Demographic questions were included at the end of the survey.
Data Analysis.
Data analysis was conducted using SPSS (version 26; IBM Corp). Descriptive statistics including mean, standard deviation, and frequency values were analyzed. The content validity ratio (CVR) was calculated for each item on the ATS-SSA instrument using a Microsoft Excel spreadsheet. The purpose of this calculation is to help identify items needing revision, deletion, or substitution.41–43 The CVR for each item was calculated using the formula:
Typically, Ne is the total number of experts who rated an item as essential.43 In this study, the Ne is the total number of participants who rated the item as clear and relevant. Here, N is the total number of participants. The CVR is a direct linear transformation from the percentage of participants rating an item as clear and relevant. If all participants rate an item as clear and relevant, the CVR = 1. If half the participants rate an item as clear and relevant, the CVR = 0. A negative CVR occurs when less than half of participants rate an item as clear and relevant.43 The minimum accepted value of the CVR is determined by the number of participants. Items with ratings below this number should be considered for revision, removal, or substitution.43 Open-response comments were initially analyzed by the principal investigator for themes and trends. A survey construction expert then analyzed the data to challenge primary investigator biases. The results were discussed to reach a consensus. All comments were considered and used to support decisions regarding revision or removal of survey items.
Results.
Nineteen electronic surveys were completed (30% response rate). Participants included athletic training educators (n = 5), athletic training preceptors (n = 7), and students currently enrolled in a graduate-level, CAATE-accredited athletic training education program (n = 7). Table 2 summarizes participant demographics. According to Ayre and Scully,42 if the number of participants rating the survey is 19, the minimum CVR should be 0.474. All but 1 question had a CVR over 0.474. All 65 items had a positive CVR. This means at least half the respondents rated each item as clear and relevant. Table 3 shows the distribution of values for survey items. Most items were rated as clear and relevant by respondents. Questions with a CVR below the minimum and questions receiving a rating of 2 (clear but not relevant) or 3 (not clear or don’t understand) were reexamined. Twenty-three comments were provided by participants. Based on results, 1 question was removed, and 2 questions were updated to improve clarity.
The updated ATS-SSA instrument includes 64 questions across 11 soft-skill themes. See Table 4 for the number of items in each theme. As before, 3 nearly identical versions of the instrument included 1 for each target audience (student, preceptor, and instructor). The instruments only vary slightly in wording, where appropriate. A 5-point Likert scale (1 = always, 5 = never) was adopted to measure the frequency each observed behavior. The updated survey was reviewed by a survey construction expert. No changes were recommended. Themes and questions used in the student version of the survey can be found in the Appendix.
The intention of the ATS-SSA instrument is to provide formative feedback to students. The result of the assessment will be an individualized document showing 3 scores for each theme. Students will be able to compare their self-ratings to those of their preceptor(s) and instructors. It is not the intention of the instrument to provide an overall composite score for all themes combined.
Stage 3: External Stage: Pilot Test
The objectives in this stage included (1) pilot testing the final ATS-SSA instrument and (2) collecting data to provide evidence of reliability or interval validity. One research question was associated with this stage: (1) How closely related are the behaviors within a theme? Pilot testing was conducted using 2 samples.
Administration and Reliability of the First Sample
The ATS-SSA instrument includes 64 questions representing 11 themes. A convenience sample of students, preceptors, and instructors from a CAATE-accredited entry-level graduate athletic training program participated in this pilot test. All participants were recruited via e-mail. Data were collected at the end of the spring 2021 semester. Instructors and preceptors had worked with the student(s) they were rating for a full semester (approximately 14 weeks). Participants included 22 students, 22 preceptors, and 3 instructors. Table 5 summarizes participant demographics. Students rated themselves on the ATS-SSA instrument. Each student was rated by 1 preceptor and by at least 1 professor.
Data analysis was conducted using SPSS. Internal consistency was examined with Cronbach α and interitem correlations. Cronbach α coefficients exceeding 0.70 and/or mean interitem correlations exceeding 0.20 are considered satisfactory for inclusion.44–46 A Cronbach α value over 0.90 may indicate redundant items, suggesting the instrument could be shortened.47 Mean interitem correlations over 0.60 may also indicate that items may not be sufficiently unique.48 These guidelines were used to identify items that should be further examined and considered for elimination or revision.
Cronbach α and mean interitem correlations were satisfactory for each theme with 1 exception. Table 6 presents all internal consistency data for each theme. The soft-skill theme of Dependable and Responsible produced a mean interitem correlation of 0.41 but a Cronbach α of 0.66. Analysis showed that removal of 1 item would increase Cronbach α to 0.73. However, because this item was perceived to be meaningful and the theme met the mean interitem correlation requirement, the item was not removed. Based on the results of the reliability analysis, all questions and soft-skill themes remained included in the survey.
The Decisive and Confident and the Growth Mindset and Action Oriented themes showed both a Cronbach α value over 0.90 and a mean interitem correlation over 0.60. These soft-skill themes were further examined. The highest correlation (0.81) in the Decisive and Confident theme occurred between 2 items (staying calm in difficult situations and making quick decisions when necessary). Three items with high correlations (above 0.82) occurred in the Growth Mindset and Action Oriented theme. These questions are related to accepting constructive criticism, viewing feedback as positive, and using feedback to make a change. Although a statistical reason to drop an item may exist, based on the purpose of the survey, a practical reason to keep them remained, such as providing more detailed information to assist in creating meaningful remediation. Additionally, due to the small sample size, the decision was made to analyze the items again when used on a second sample.
Administration and Reliability of the Second Sample
A convenience sample of students, preceptors, and instructors from a CAATE-accredited, entry-level graduate athletic training program participated in this stage. All participants were recruited via e-mail. Data were collected at the end of the fall 2021 semester. Participants included 28 students, 32 preceptors, and 3 instructors. Fourteen students in this sample also participated in the first stage 3 pilot test. Instructors and preceptors had worked with the student(s) they were rating for a full semester (approximately 14 weeks). All ratings were based on the current semester. Table 7 summarizes participant demographics. Students rated themselves on the ATS-SSA instrument. Each student was rated by 2 professors and at least 1 preceptor. Data analysis was conducted using the same criteria as the first sample. All themes produced a minimum Cronbach α of 0.70 and average interitem correlation of 0.20. No themes produced a Cronbach α value over 0.90 or a mean interitem correlation over 0.60. Based on the results of the reliability analysis, all questions and soft-skill themes were included in the final instrument. Table 6 presents all internal consistency data for each theme.
DISCUSSION
The purpose of this study was to develop and validate a new multisource instrument to assess soft skills in master’s level professional athletic training students. The ATS-SSA instrument was developed using a multistage process of development and validation. The stages involved a literature search, survey item development, and review by athletic training educators, preceptors, and entry-level athletic training students. Each survey item was extensively reviewed, and revisions were conducted to minimize measurement error. Survey length and structure was thoughtfully considered to optimize utility. The use of the rigorous content validation approach adopted in this study provided evidence that the instrument was comprehensive, clear, and complete enough to establish the tool’s credibility in the preliminary stages. The final survey instrument consists of 11 soft-skill themes and a total of 64 questions. To the author’s knowledge, this survey is unique in its focus and design.
Implications
The design of the ATS-SSA instrument allows students to receive formative feedback from both clinical and academic sources. Combined with their self-assessments, students should be able to see how well their self-perceptions match with those of their preceptor(s) and instructors. This will allow students to identify areas where they overappraise or underappraise their soft skills, raising self-awareness. This tool may also be used to help program directors to direct efforts and resources effectively while creating more meaningful and targeted instruction. When supported with guidance, reflection, mentoring, and suggestions for remediation, this instrument could be a powerful tool to assist in positive behavior change. Furthermore, this tool can be used multiple times over the length of a students’ academic program.
Limitations
The use of nonrandomized sampling in stages 2 and 3 represents a limitation in this study. In stage 2, a convenience sample was used to obtain opinions from educators and preceptors working in a variety of settings across the country. All students and some preceptors from the author’s home institution were recruited in this stage. Although responses were anonymous, bias may result due to the participant recruitment method. In stage 3, the 2 pilot tests were conducted using a small convenience sample of athletic training educators, preceptors, and students from 1 private Midwestern university Master of Science in Athletic Training Program. This is the author’s home institution and program. Some participants were used in both stage 3 pilot tests. Additionally, some students and preceptors participated in both stages 2 and 3. The individuals participating in this study may or may not be representative of a larger population. Additionally, the results of stage 3 of this study are presented from the author’s home institution only. If this instrument is to be adopted by other athletic training programs, further reliability analysis should be completed. The author encourages confirmation of findings from other universities and Master of Science in Athletic Training programs. Also, given the small sample size, it is not appropriate to conduct a factorial analysis to further analyze reliability.
CONCLUSIONS AND RECOMMENDATIONS
The outcome of this study is a multisource soft skills assessment tool. It can be used to provide master’s level professional athletic training students with formative feedback. Future researchers should explore student perceptions of the tool and perceived utility of the feedback. Future researchers should also examine the efficacy of athletic training educators using this instrument to design targeted interventions to help students improve the quality of select soft skills.
Study flow chart.
Contributor Notes