Automated Quantification of the Landing Error Scoring System With a Markerless Motion-Capture System
The Landing Error Scoring System (LESS) can be used to identify individuals with an elevated risk of lower extremity injury. The limitation of the LESS is that raters identify movement errors from video replay, which is time-consuming and, therefore, may limit its use by clinicians. A markerless motion-capture system may be capable of automating LESS scoring, thereby removing this obstacle. To determine the reliability of an automated markerless motion-capture system for scoring the LESS. Cross-sectional study. United States Military Academy. A total of 57 healthy, physically active individuals (47 men, 10 women; age = 18.6 ± 0.6 years, height = 174.5 ± 6.7 cm, mass = 75.9 ± 9.2 kg). Participants completed 3 jump-landing trials that were recorded by standard video cameras and a depth camera. Their movement quality was evaluated by expert LESS raters (standard video recording) using the LESS rubric and by software that automates LESS scoring (depth-camera data). We recorded an error for a LESS item if it was present on at least 2 of 3 jump-landing trials. We calculated κ statistics, prevalence- and bias-adjusted κ (PABAK) statistics, and percentage agreement for each LESS item. Interrater reliability was evaluated between the 2 expert rater scores and between a consensus expert score and the markerless motion-capture system score. We observed reliability between the 2 expert LESS raters (average κ = 0.45 ± 0.35, average PABAK = 0.67 ± 0.34; percentage agreement = 0.83 ± 0.17). The markerless motion-capture system had similar reliability with consensus expert scores (average κ = 0.48 ± 0.40, average PABAK = 0.71 ± 0.27; percentage agreement = 0.85 ± 0.14). However, reliability was poor for 5 LESS items in both LESS score comparisons. A markerless motion-capture system had the same level of reliability as expert LESS raters, suggesting that an automated system can accurately assess movement. Therefore, clinicians can use the markerless motion-capture system to reliably score the LESS without being limited by the time requirements of manual LESS scoring.Context:
Objective:
Design:
Setting:
Patients or Other Participants:
Main Outcome Measure(s):
Results:
Conclusions:
Lower extremity musculoskeletal injuries among athletes1,2 and military personnel3 result in substantial medical costs, forced attrition from physical activity,1,2,4 and long-term physical5,6 and financial consequences.7 Therefore, they are a primary concern for athletic coaches, military commanders, and health care professionals.8 It is essential to identify factors that increase injury risk so that targeted injury-intervention strategies may be implemented.9,10
Whereas the risks for lower extremity injury are multifactorial,8 1 primary predictor of noncontact injury is lower extremity biomechanical patterns.8,11–14 Laboratory-based movement assessments effectively identify high-risk biomechanical patterns,12–14 but they are largely inaccessible to sports medicine clinicians, especially in large group settings. Cost-effective, field-expedient movement assessments are needed so that clinicians can quickly and accurately identify individuals at greater injury risk.8,11,15 The Landing Error Scoring System (LESS) is one such movement assessment.11,15
The LESS is a valid and reliable movement assessment15 that meets many of the requirements presented by a consortium8 of civilian and military experts on injury risks and prevention, but it has limitations. It requires individuals to complete a jump-landing movement assessment while being videotaped from the frontal and sagittal planes. The videos are played back at a later date and manually scored by trained raters.15 Given the time requirements of the LESS, sports medicine clinicians may be impeded from implementing it as part of their preparticipation and return-to-activity assessments.8
Therefore, the consortium of military and civilian experts has called for automated systems that accurately and quickly identify individuals at increased injury risk.8,16 Automated systems substantially reduce the time required to screen individuals for injury risks.16 Automated markerless motion-capture systems can capture and score full-body kinematics without using reflective markers or electromagnetic sensors. Other markerless motion-capture systems have produced reliable and valid measures of trunk and lower extremity joint kinematics during functional movement assessments.17–20 However, these systems have not been validated for scoring the LESS. Therefore, the purpose of our study was to determine the reliability of an automated markerless movement-assessment system that can be used to quickly and efficiently screen large groups of individuals.
METHODS
Participants
A total of 57 healthy, physically active individuals (47 men, 10 women; age = 18.6 ± 0.6 years, height = 174.5 ± 6.7 cm, mass = 75.9 ± 9.2 kg) completing a US Military Academy Cadet Basic Training course participated in this study. All US Military Academy cadets are required to regularly complete physical activity as part of the Cadet Basic Training course. The participants were selected randomly from a larger prospective study population in which lower extremity movement patterns and injury risks were examined. The Keller Army Community Hospital Institutional Review Board approved the study, and all participants provided written informed consent.
Procedures
Participants wore their own athletic shoes and standardized shorts and T-shirts during all testing procedures. They successfully completed 3 trials of a jump-landing movement assessment after a minimum of 1 practice trial. For the jump-landing assessment, participants jumped from a 30-cm-tall box to a designated area located 0.9 m in front of the box. We instructed them to complete a maximal vertical jump immediately after landing in the designated area. A trial was deemed successful if the participant (1) jumped off the box with both feet leaving the box at the same time; (2) jumped forward, and not vertically, to reach the designated area; (3) landed with both feet in the designated area; (4) jumped vertically, and not forward, during the maximal jump; and (5) completed the movement in a fluid motion.15
Jump-landing trials were simultaneously recorded by 2 (frontal- and sagittal-plane views) standard video cameras (model DCR-HC30; Sony Corp of America, Park Ridge, NJ) and a depth camera (frontal-plane view only; Kinect sensor, version 1; Microsoft Corp, Redmond, WA). The depth camera was controlled by a standard laptop computer. After data collection, 2 expert raters (T.C.M., L.E.S.) independently evaluated the standard video data and scored the LESS.15 Athletic Movement Assessment software (PhysiMax Technologies Ltd, Tel Aviv, Israel) was used to evaluate the depth-camera data and score the LESS (Figure 1).
Citation: Journal of Athletic Training 52, 11; 10.4085/1062-6050-52.10.12
Main Outcome Measures
The LESS is a standardized scoring rubric used to visually identify aberrant lower extremity and trunk kinematics during a jump-landing assessment. Items on the LESS are evaluated at initial ground contact and peak knee flexion and during the time between initial ground contact and peak knee flexion.15 A larger total LESS score indicates more aberrant kinematics than a smaller total score. The LESS is a valid 2-dimensional assessment of lower extremity and trunk kinematics and has excellent intrarater (intraclass correlation coefficient [ICC] (2,1) = 0.91, SEM = 0.42) and good interrater (ICC [2,k] = 0.84, SEM = 0.71) reliability.15 It can discriminate between individuals with high-risk (ie, aberrant) and low-risk kinematics.11,21 The LESS scoring rubric that we used was expanded from the original 17 items to 22 items, increasing the original total LESS score from 19 to 24. The 5 additional LESS items included further clarification of asymmetric foot contact (1 item each for timing and plantar flexion), excessive trunk-flexion displacement, asymmetric weight shift, and knee “wobble.”22 The markerless motion-capture system that we used was incapable of automatically scoring the Overall impression item on the LESS, so this item was not analyzed. Therefore, the total LESS score for this study was 22.
The expert raters independently completed standardized LESS training as described by Padua et al.15 Before data collection for our study began, each expert rater had scored more than 250 participants (750 individual jump-landing trials) using the LESS. They independently scored the 57 participants included in this study. The expert-rater datasets were compared, and discrepancies between them were identified. For a single LESS item with a discrepancy, the expert raters watched the video trial again and agreed on the presence or absence of the error. If they could not agree on the item, an additional expert rater (B.S.F.) was consulted to make the final decision. The final composite dataset, with the agreed-upon scores, was deemed the consensus expert score and was used for comparison with the LESS scores that the markerless motion-capture system provided (Figure 1).
The markerless motion-capture system automatically captured and scored full-body kinematics. The system uses cloud-based technology to process the depth-camera data via proprietary kinematic machine-learning algorithms. The algorithms extract, track, and dynamically refine virtual markers on the individual's body to assess dynamic motion. The algorithms can calculate kinematic variables, including joint angles, ranges, velocities, and accelerations. Similar systems have been validated against standard marker-based systems.17–20
Data Analysis
For an error to be included in the final dataset, the participant had to score the error on a single LESS item on at least 2 of 3 trials. We calculated κ and prevalence- and bias-adjusted κ (PABAK) statistics and percentage agreement for each LESS item. Interrater reliability between the expert LESS raters was evaluated. After establishing the consensus expert scores, we evaluated interrater reliability between the consensus expert scores and the markerless motion-capture system's LESS scores. Paired-samples t tests were used to compare total LESS scores between the expert raters and between the consensus expert scores and the markerless motion-capture system.
We included PABAK statistics due to the nature of the data. Traditional κ statistics are influenced by the prevalence of a finding (LESS item errors); if the prevalence of a positive finding is either very high or very low (high prevalence index), the chance of agreement is also very high, which reduces the value of the κ statistic.23 Similarly, rater bias influences κ statistics. Rater bias increases if 2 raters disagree asymmetrically about the proportion of positive findings (eg, 1 rater identifies errors more frequently than a second rater for a specific LESS item).23 Prevalence and bias should be considered when examining and interpreting κ statistics.23 We interpreted the κ and PABAK statistics as follows: high reliability (0.81–1.00), good reliability (0.61–0.80), moderate reliability (0.41–0.60), fair reliability (0.21–0.40), and poor reliability (≤0.20).24
All statistical analyses were completed using PASW Statistics for Windows (version 21.0; SPSS Inc, Chicago, IL). The α level was set a priori at ≤.05.
RESULTS
Agreement existed between the expert raters for 15 of the 21 LESS items (Figure 2). We observed perfect agreement between the expert raters for hip flexion at initial contact and hip-flexion displacement. The PABAK statistics were equal to or greater than the κ statistics for all but 5 LESS items: lateral trunk flexion at initial contact, knee-valgus angle at initial contact, narrow stance width, toe-out foot position, and asymmetric weight shift. The total LESS scores also differed between the expert raters (T.C.M. = 6.25 ± 1.92, L.E.S. = 3.79 ± 2.12; t56 = 4.75; P < .001). Prevalence and bias indices are reported in the Table.
Citation: Journal of Athletic Training 52, 11; 10.4085/1062-6050-52.10.12
Similar agreement was found between the consensus expert scores and the markerless motion-capture system for 14 of the 21 LESS items (Figure 3). We observed perfect agreement between the consensus expert scores and the markerless motion-capture system for hip flexion at initial contact, asymmetric foot contact: plantar flexion, wide stance width, toe-in foot position, and hip-flexion displacement. The PABAK statistics were equal to or greater than the κ statistics for all LESS items except lateral trunk flexion at initial contact and maximum knee valgus. The total LESS scores differed between the 2 scoring systems (consensus expert scores = 4.64 ± 1.88, markerless motion-capture system = 5.84 ± 2.03; t56 = 3.89, P < .001). Prevalence and bias indices are reported in the Table.
Citation: Journal of Athletic Training 52, 11; 10.4085/1062-6050-52.10.12
DISCUSSION
The major finding of our study was that the markerless motion-capture system was as reliable in identifying movement errors during the LESS as 2 expert raters. This finding has important clinical implications because one of the greatest barriers to widespread implementation of movement assessments is the time needed to collect and analyze the data. A reliable, automated version of a validated lower extremity movement assessment will save substantial time16 and provide clinicians with greater access to clinical movement assessments.
Overall, the markerless motion-capture system had moderate reliability compared with a consensus of expert LESS raters (average κ [κavg] = 0.48 ± 0.40; Figure 3). When the PABAK statistics were evaluated, it was evident that good reliability existed between the markerless motion-capture system and the expert consensus scores (average PABAK [PABAKavg] = 0.71 ± 0.27). We believe the PABAK is the more appropriate statistic to examine for these data, given that a number of LESS items have very low prevalence (eg, hip flexion at initial contact) and, to a lesser extent, rater bias (ie, a rater “overscores” an error for a particular LESS item).
The PABAK statistics showed that the markerless motion-capture system and expert consensus scores had perfect agreement for 5 items, high agreement for 2 items, good agreement for 8 items, moderate agreement for 2 items, fair agreement for 2 items, and poor agreement for 2 items. We recognize the agreement was not perfect between the systems, but agreement was not perfect between the expert LESS raters either (κavg = 0.45 ± 0.35, PABAKavg = 0.67 ± 0.34; Figure 2). The expert LESS raters had perfect agreement for 2 items, high agreement for 7 items, good agreement for 6 items, moderate agreement for 1 item, fair agreement for 2 items, and poor agreement for 3 items. These findings are important because using expert raters to score and analyze LESS data is the current standard practice.15,25,26
Padua et al15 showed that traditional LESS scoring had excellent intrarater and good interrater reliability. However, when the LESS was scored in real time (10 items), the interrater reliability decreased slightly (ICC [2,1] = 0.72–0.81).25 The markerless motion-capture system that we used is not limited to the 10 items for real-time scoring and still maintains good reliability. In addition, the κ values that we observed were similar to those reported for interrater reliability between expert and novice raters (κ range = 0.46–0.88).26
In a recent literature review, Moran et al27 examined the reliability of another common movement assessment, the Functional Movement Screen (FMS), and reported moderate evidence of acceptable live interrater scoring of the FMS (κ ≥ 0.4). Novice raters had moderate to high reliability (κrange = 0.54–1.00; κavg = 0.74 ± 0.15), with associated percentage agreement ranging from 74.4% to 100.0%, whereas expert FMS raters had slightly lower reliability (κrange = 0.40–0.95; κavg = 0.68 ± 0.17), with associated percentage agreement ranging from 69.2% to 97.4%.28 When expert and novice FMS raters were compared, levels of reliability (κrange = 0.31–1.0; κavg = 0.81 ± 0.22) and percentage agreement (80%–100%) were similar.29,30 The reliability observed for the FMS in these studies has been deemed “acceptable,”27 and the reliability of the markerless motion-capture system that we used is comparable.
The differences in total LESS scores between the expert raters (2.46 points; 11.71% of maximum possible score) and between the expert consensus scores and the markerless motion-capture system (1.20 points; 5.71% of the maximum possible score) were larger than what has been reported in the literature. Padua et al25 examined the reliability of LESS real-time scoring and found a difference of 0.4 points (2.67% of maximum possible score) between raters scoring the same individuals. Furthermore, individuals who went on to sustain a noncontact anterior cruciate ligament injury scored 1.82 points (9.58% of maximum possible score) higher than individuals who did not sustain such an injury.11 If the threshold for meaningful clinical differences in LESS scores is conservatively set at a difference equal to 9% of the maximum possible LESS score, then the differences in scores between the expert consensus scores and the markerless motion-capture system fell well within this threshold. However, the differences between the expert LESS raters' total scores did not. This was a surprising and potentially unfortunate finding for the expert LESS raters. It supports the use of the automated markerless motion-capture system because the system can provide LESS scores that are clinically comparable with scores reported by the current criterion standard, consensus expert scoring.
Trained LESS raters require 3 to 4 minutes to score 3 jump-landing trials.15 If traditional LESS scoring is implemented for a collegiate basketball team (15 athletes), a minimum of 45 minutes would be required to score the entire team. This duration does not include the time needed to load the videos to a computer and mark them so they may be efficiently scored. The markerless motion-capture system that we used can analyze 3 jump-landing trials almost instantaneously in real time. The only time requirement for this system is actually recording the jump-landing trials, which takes no more time than traditional LESS scoring. If this same rationale is expanded to implementing the markerless motion-capture system for a brigade of army personnel (3500 individuals),8 the time saved is greatly magnified. Traditional LESS scoring would take at least 10 500 minutes (175 hours), whereas automated LESS scoring would take only the time required to collect the jump-landing trials. Automation would also make the scores immediately available, which would allow individualized injury-prevention programs to be implemented earlier in the athletic season or military training.
The automated markerless motion-capture system that we used is publicly available for purchase. The costs of the actual data-acquisition equipment (Microsoft Kinect camera and laptop) are comparable with the costs of purchasing the equipment necessary to complete traditional LESS scoring (2-dimensional video cameras and computer). Additional costs are associated with purchasing the proprietary software used to analyze the markerless motion-capture system data. However, they could potentially be offset because it would no longer be necessary to manually score the jump-landing trials. Both of the aforementioned LESS scoring methods are substantially less expensive than traditional laboratory-based motion-capture systems. Furthermore, we have successfully trained individuals with no previous experience to collect data on and export it from the markerless motion-capture system in less than 15 minutes. This is a substantial time savings compared with the many hours required to train individuals to accurately score the LESS or collect, reduce, and analyze data using traditional laboratory-based motion-capture systems.
In addition to the time and potential cost savings that the markerless motion-capture system provides, it has 2 other important benefits. First, it has fixed algorithms and consistently scores jump-landing trials. This is useful when conducting serial testing to determine whether an individual's rehabilitation or injury-prevention program is working to improve kinematics.8 Furthermore, expert raters may not always agree on LESS errors. Our data showed that the mean difference between expert raters for total LESS scores was relatively large (2.46). This disagreement could artificially increase or decrease serial LESS scores and, thus, incorrectly influence the implementation of injury-prevention strategies. The fixed algorithms of the automated system eliminate the need to have the same individual score serial jump-landing trials. Second, the markerless motion-capture system does not become “fatigued” when scoring jump-landing trials, which minimizes potential errors in the data due to the LESS rater's losing focus.
Movement assessments are beneficial only if they result in usable information. Therefore, a consortium8 of military and civilian experts on lower extremity injury and injury-prevention strategies has called for automated individualized reports on movement assessments so that clinicians can develop and implement injury-prevention strategies.16 The markerless motion-capture system automatically compiles these reports at the end of each testing session. The reports include a summary of the individual LESS items, the total LESS score, and pictures of the participant throughout the jump-landing trials. The information provided in these reports enables clinicians to quickly and efficiently develop efficacious injury-prevention and rehabilitation strategies.
Our study had limitations. First, the markerless motion-capture system was incapable of automatically scoring the Overall impression item on the LESS, so we did not analyze this LESS item. The most recent version of the software allows the individual operating the system to score this item in real time (at the time of data collection). The Overall impression item is reliable between raters30 and testing sessions.15,25 We believe the real-time interaction between the clinician and the markerless motion-capture system is a strength of the system. Second, the system was reliable for a group of physically active military cadets, who may not be similar to other physically active individuals or athletes. Third, this markerless motion-capture system has not been validated against the criterion standard of movement assessments, 3-dimensional motion-capture software. However, similar markerless motion-capture systems have been validated.17–20 Therefore, we are confident that the markerless motion-capture system can accurately identify the gross differences in lower extremity and trunk-movement patterns that are scored by the LESS.
CONCLUSIONS
An automated markerless motion-capture system was as reliable as the current criterion standard of LESS scoring by expert raters. The markerless motion-capture system allows clinicians to conduct lower extremity movement assessments en masse and have reports autogenerated so that individuals at greater risk of noncontact lower extremity injuries can be identified. Clinicians can use these reports to develop individualized programs and rehabilitation protocols, which have been shown to decrease the risk of injury.9,10 The long-term benefits of reducing injury risks may greatly improve the overall health and well-being of physically active civilians and military personnel.
Flow chart of data collection and reduction. a PhysiMax Athletic Movement Assessment software (PhysiMax Technologies Ltd, Tel Aviv, Israel).
Comparison between expert rater 1 and 2 scores for A, initial contact, and B, maximum and displacement items of the Landing Error Scoring System. a represents a significant κ value (α ≤ .05).
Comparison between markerless motion-capture system and expert-rater consensus scores for A, initial contact, and B, maximum and displacement items of the Landing Error Scoring System. a represents a significant κ value (α ≤ .05).
Contributor Notes
Dr Mauntel is now in the Department of Orthopaedics, Walter Reed National Military Medical Center, Bethesda, MD