Three-dimensional motion tracking correlates with skill level in upper gastrointestinal endoscopy

• Introduction
• Methods
• Participants
• Data collection
• Measures
• Data analyses
• Results
• Discussion
• References

Background and study aim: Feedback is an essential part of training in upper gastrointestinal endoscopy. Virtual reality simulators provide limited feedback, focusing only on visual recognition with no feedback on the procedural part of training. Motion tracking identifies patterns of movement, and this study aimed to explore the correlation between skill level and operator movement using an objective automated tool.

Methods: In this medical education study, 37 operators (12 senior doctors who performed endoscopic retrograde cholangiopancreatography, 13 doctors with varying levels of experience, and 12 untrained medical students) were tested using a virtual reality simulator. A motion sensor was used to collect data regarding the distance between the hands, and height and movement of the scope hand. Test characteristics between groups were explored using Kruskal-Wallis H and Man-Whitney U exact tests.

Results: All motion-tracking metrics showed discriminative ability primarily between experts and novices in specific sequences.

Conclusion: Motion tracking can discriminate between operators with different experience levels in upper gastrointestinal endoscopy. Motion tracking can be used to provide feedback regarding posture and movement during endoscopy training.

Introduction

The use of virtual reality in gastrointestinal endoscopy training has been established as a beneficial method for early training of technical skills within a risk-free environment [1]. Feedback is essential in training and a practical skill is best acquired when the learner receives feedback on the procedural part of the training [2] [3]. Virtual reality simulators only set goals and provide feedback on outcomes of the procedure [2], which could explain why current virtual reality simulators for upper gastrointestinal endoscopy (UGE) only allow novices to acquire basic skill sets [1] [4]. Expert guidance is typically utilized to evaluate procedural performance, but this approach requires resources and is prone to rater bias [2] [5]. Therefore, objective automated measures of process goals could potentially improve goal setting and feedback, and could increase the value of self-guided learning by addressing the shortcomings of the current virtual reality training simulators.

Motion tracking during a technical procedure can identify patterns of movement specific to the procedure. Cotton described the optimal stance for UGE, where the instrument “runs in a gentle curve to the patient’s mouth” and “torque applied to the straightened shaft is an important part of steering” [6]. To obtain a gentle curve, the operator’s hands must be wide apart and the hand holding the scope (the scope hand) must be held high. Furthermore, in order to achieve fluency of the procedure, the scope hand should be moved in an ergonomic fashion [5] and “be precise in movements” [6].

The aim of the present study was to determine whether specific operator movements correlate with skill level, using motion tracking as an objective automated tool.

Methods

All UGEs were performed using the validated GI Mentor II virtual reality simulator (Simbionix USA Corp., Cleveland, Ohio, USA) [7] [8]. The movements of the operators were tracked using the Microsoft Kinect motion sensor, a system that was originally designed for the Xbox gaming platform (Microsoft Corp., Redmond, Washington, USA). The sensors create a three-dimensional map of reflections from the object in the scene, which is used to provide skeletal movement tracking. The accuracy of the Microsoft Kinect is 1 – 2 cm at a distance to the operator of 1 – 2.9 m [9]. The Microsoft Kinect was placed above the GI Mentor II and pointed at the chest of the operator. The resulting three-dimensional axes are illustrated in [Fig. 1]. Mapping was done at 30 frames per second.

All operators completed two procedures. The motion sensor recorded two sequences in each procedure. Sequenced recording was introduced in order to separate the different phases of the procedure and identify specific movements. The first sequence was from the pharynx to the angulus in the stomach. The second sequence was from the pyloric antrum through the pyloric sphincter to the papilla.

Participants

A total of 37 operators with three different skill levels were included: novices (n = 12) were medical students with no experience in UGE; intermediates (n = 13) were doctors with clinical experience in UGE; and experts (n = 12) were doctors trained in endoscopic retrograde cholangiopancreatography, which in Denmark requires vast experience in upper and lower gastrointestinal endoscopy procedures ([Table 1]). Participants were volunteers from the population of medical students and doctors in the Capital and Zealand regions of Denmark. Participating surgeons and gastroenterologists were from eight different hospitals.

Data collection

All participants signed an informed consent form and provided demographic information and details of experience. Novices were instructed on how to hold the scope, the basic functions of the instrument, and retroflection of the scope, as described in textbook material [6]. All participants received a short introduction to the virtual reality simulator and performed a warm-up procedure using the device. The cutoff time was 10 minutes, and the total test time 20 minutes. Two researchers tested all participants.

Measures

The operators were monitored on “distance between the hands” (average distance between the left and the right hand), “height of the scope hand” (average distance between the left shoulder [origin] and the left hand), and “distance moved with the scope hand” (total distance moved with the left hand frame by frame). These measures were defined from prior experience with the motion sensor and textbook descriptions of the optimal stance [6] [10].

Data analyses

All data were described as median and range. Comparisons among groups were performed using the Kruskal-Wallis H exact test (significance level 0.05). Pair-wise comparisons were performed using the Mann-Whitney U exact test. To account for multiple testing, the significance level was set to 0.01.

Statistical analysis was performed using IBM SPSS Statistics for Macintosh statistical software package (Version 22.0; IBM Corp., Armonk, New York, USA). Data graphics were constructed using Python (Python Software Foundation, Python Language Reference, version 2.7. http://www.python.org).

In accordance with the Danish Committee Act § 1 cl. 4, the Research Ethics Committee of the Capital Region of Denmark waived the need for full ethical approval (Case no: H-3-2014-FSP25).

Results

Testing was conducted between 10 March and 5 May 2014.

The median distance between the hands was different in the three groups ([Fig. 2]). The differences were significant for the first part of the procedure (passage through esophagus), with P = 0.02 and P = 0.04 for procedures 1 and 2, respectively. Experts kept their hands approximately 10 cm wider apart than novices (P = 0.01 on post hoc tests). [Fig. 3] shows the relationship between skill level, mean distance, and SD as a percentage of total procedure time.

The median height of the scope hand was significantly different for the first part of procedure 1 (P = 0.05) and both parts of procedure 2 (P = 0.02 for both parts). Experts kept their scope hand 9 cm higher than novices and the differences were statistically significant for passing the pylorus (P < 0.01). Experts kept their scope hand 11 cm higher than intermediates in procedure 2 when passing through the esophagus (P = 0.01).

The median distance moved by the scope hand was different in the three groups. This was significant for passing through the pylorus in procedures 1 and 2 (P = 0.01 and P < 0.01, respectively). Post hoc tests showed that novices moved their hand 169 – 382 cm more than experts (P < 0.01) and intermediates (P = 0.01).

Comparison of the motion-tracking metrics between the three operator groups is shown in supplementary Table e2, Table e3, and Table e4. Intergroup differences are shown in supplementary Table e5. Motion-tracking metric data from individual operators are shown in supplementary Table e6, Table e7, and Table e8. All supplementary tables are available online.

Discussion

The movement metrics identified can be used to distinguish between novices and experts and to help identify the stance of skilled endoscopists, who generally have their hands far apart and their scope hand high and steady. A focus on stance may improve procedure performance. The motion sensor is an objective automated tool that can be implemented easily in both virtual reality training and in the endoscopy suite.

Keeping the hands wider apart and the scope hand high will provide a straight scope. A straight scope allows superior control over the tip of the scope, greater maneuvering ability, and prevents excessive loop formation in the greater curvature [6]. When the scope passes through the pyloric sphincter, most of the rest of the scope is inside the patient; consequently, hands will gradually come closer and the difference between novice and expert is eliminated on average. [Fig. 3] showed that, in the second sequence, experts begin with their hands apart, but the gap to the novice group narrows as they approach the duodenum. The SD, as illustrated by the colored band around the mean, is wider in the novice group and decreases with skill level. This implies that experts consistently operate with hands wider apart. An outlier in the expert group was a short woman, who consistently had her hands close together, although for her they were relatively wide apart. Therefore, future studies should adjust for height.

Keeping the scope straight is essential, especially when passing through the pylorus. Unskilled operators who do not position the scope correctly may use unnecessary force [5] [6], which endangers the patient and impedes advancement of the scope.

In clinical procedures, an expert will display economy of movement [5]. Often operators will spend an entire day in the endoscopy suite, and consequently a focus on ergonomics is essential to minimize work-related injuries. The metric distance moved with the scope hand shows evidence of an optimal movement pattern that novices could benefit from pursuing. Because novices take more time to complete each sequence, the metric should be interpreted with caution.

One strength of this study is that doctors from eight different hospitals and two different specialties participated. Therefore, generalization is more plausible. The study defines objective metrics based on motion tracking. One dilemma in assessment is rater bias, which accounts for a substantial part of an assessment score [11]. Use of the motion sensor completely eliminates rater bias when analyzing data.

One limitation of this study is that even though a sample size of 37 is considered large in medical education research, the sample size is small in absolute terms [12]. Therefore, caution is advised when interpreting the results. In addition, the participants volunteered for study inclusion, which makes the data vulnerable to volunteer bias [13]. Complication rates are low in UGE; therefore, large sample sizes are required to show significant differences in patient clinical outcomes, which is rarely possible in a medical education study [14]. Learning outcomes could be tested in future studies as an intervention study comparing training using virtual reality simulator metrics with motion-tracking metrics.

In conclusion, three process goals were identified in UGE through motion-tracking analyses. A correlation was identified between operator movement and skill level. Further research is needed to determine whether motion analysis could have a positive impact on learning and patient outcome.

Competing interests: None

• References

• 1 Walsh CM, Sherlock ME, Ling SC et al. Virtual reality simulation training for health professions trainees in gastrointestinal endoscopy. Cochrane Database of Systematic Reviews 2012; 6: CD008237
  PubMedSearch in Google Scholar
• 2 Brydges R, Carnahan H, Safir O et al. How effective is self-guided learning of clinical technical skills? It’s all about process. . Med Educ 2009; 43: 507-515
  CrossrefPubMedSearch in Google Scholar
• 3 Magill RA. Motor learning and control. 8th edn New York: McGraw-Hill; 2006
  Search in Google Scholar
• 4 Ferlitsch A, Schoefl R, Puespoek A et al. Effect of virtual endoscopy simulator training on performance of upper gastrointestinal endoscopy in patients: a randomized controlled trial. Endoscopy 2010; 42: 1049-1056
  Thieme ConnectPubMedSearch in Google Scholar
• 5 Martin JA, Regehr G, Reznick R et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997; 84: 273-278
  CrossrefPubMedSearch in Google Scholar
• 6 Cotton PB, Williams CB. Practical gastrointestinal endoscopy. 6th edn. Chichester: Wiley-Blackwell; 2011
  Search in Google Scholar
• 7 Moorthy K, Munz Y, Jiwanji M et al. Validity and reliability of a virtual reality upper gastrointestinal simulator and cross validation using structured assessment of individual performance with video playback. Surg Endosc 2004; 18: 328-333
  CrossrefPubMedSearch in Google Scholar
• 8 Koch AD, Buzink SN, Heemskerk J et al. Expert and construct validity of the Simbionix GI Mentor II endoscopy simulator for colonoscopy. Surg Endosc 2008; 22: 158-162
  CrossrefPubMedSearch in Google Scholar
• 9 Mobini A, Behzadipour S, Saadat FoumaniM. Accuracy of Kinect’s skeleton tracking for upper body rehabilitation applications. Disabil Rehabil Assist Technol 2014; 9: 344-352
  CrossrefPubMedSearch in Google Scholar
• 10 Svendsen MB, Preisler L, Hillingsoe JG et al. Using motion capture to assess colonoscopy experience level. World J Gastrointest Endosc 2014; 6: 193-199
  CrossrefPubMedSearch in Google Scholar
• 11 Konge L, Vilmann P, Clementsen P et al. Reliable and valid assessment of competence in endoscopic ultrasonography and fine-needle aspiration for mediastinal staging of non-small cell lung cancer. Endoscopy 2012; 44: 928-933
  Thieme ConnectPubMedSearch in Google Scholar
• 12 Cook DA, Hatala R. Got power? A systematic review of sample size adequacy in health professions education research. . Adv Health Sci Educ Theory Pract 2015; 20: 73-83
  CrossrefPubMedSearch in Google Scholar
• 13 Downing SM, Yudkowsky R. Assessment in health professions education. 1st edn. New York: Routledge; 2009
  Search in Google Scholar
• 14 Cook DA, West CP. Perspective: reconsidering the focus on “outcomes research” in medical education: a cautionary note. Acad Med 2013; 88: 162-167
  CrossrefPubMedSearch in Google Scholar

Corresponding author

• References

• 1 Walsh CM, Sherlock ME, Ling SC et al. Virtual reality simulation training for health professions trainees in gastrointestinal endoscopy. Cochrane Database of Systematic Reviews 2012; 6: CD008237
  PubMedSearch in Google Scholar
• 2 Brydges R, Carnahan H, Safir O et al. How effective is self-guided learning of clinical technical skills? It’s all about process. . Med Educ 2009; 43: 507-515
  CrossrefPubMedSearch in Google Scholar
• 3 Magill RA. Motor learning and control. 8th edn New York: McGraw-Hill; 2006
  Search in Google Scholar
• 4 Ferlitsch A, Schoefl R, Puespoek A et al. Effect of virtual endoscopy simulator training on performance of upper gastrointestinal endoscopy in patients: a randomized controlled trial. Endoscopy 2010; 42: 1049-1056
  Thieme ConnectPubMedSearch in Google Scholar
• 5 Martin JA, Regehr G, Reznick R et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997; 84: 273-278
  CrossrefPubMedSearch in Google Scholar
• 6 Cotton PB, Williams CB. Practical gastrointestinal endoscopy. 6th edn. Chichester: Wiley-Blackwell; 2011
  Search in Google Scholar
• 7 Moorthy K, Munz Y, Jiwanji M et al. Validity and reliability of a virtual reality upper gastrointestinal simulator and cross validation using structured assessment of individual performance with video playback. Surg Endosc 2004; 18: 328-333
  CrossrefPubMedSearch in Google Scholar
• 8 Koch AD, Buzink SN, Heemskerk J et al. Expert and construct validity of the Simbionix GI Mentor II endoscopy simulator for colonoscopy. Surg Endosc 2008; 22: 158-162
  CrossrefPubMedSearch in Google Scholar
• 9 Mobini A, Behzadipour S, Saadat FoumaniM. Accuracy of Kinect’s skeleton tracking for upper body rehabilitation applications. Disabil Rehabil Assist Technol 2014; 9: 344-352
  CrossrefPubMedSearch in Google Scholar
• 10 Svendsen MB, Preisler L, Hillingsoe JG et al. Using motion capture to assess colonoscopy experience level. World J Gastrointest Endosc 2014; 6: 193-199
  CrossrefPubMedSearch in Google Scholar
• 11 Konge L, Vilmann P, Clementsen P et al. Reliable and valid assessment of competence in endoscopic ultrasonography and fine-needle aspiration for mediastinal staging of non-small cell lung cancer. Endoscopy 2012; 44: 928-933
  Thieme ConnectPubMedSearch in Google Scholar
• 12 Cook DA, Hatala R. Got power? A systematic review of sample size adequacy in health professions education research. . Adv Health Sci Educ Theory Pract 2015; 20: 73-83
  CrossrefPubMedSearch in Google Scholar
• 13 Downing SM, Yudkowsky R. Assessment in health professions education. 1st edn. New York: Routledge; 2009
  Search in Google Scholar
• 14 Cook DA, West CP. Perspective: reconsidering the focus on “outcomes research” in medical education: a cautionary note. Acad Med 2013; 88: 162-167
  CrossrefPubMedSearch in Google Scholar

• Supplementary Material