Task Dynamics of Prior Training Influence Visual Force Estimation Ability During Teleoperation

We recruited 33 participants, between the ages of 18 and 36. All participants were right-handed and were balanced for sex over all haptic training conditions. All participants were novices in that they either had never used a da Vinci Surgical System or da Vinci Research Kit (Intuitive Surgical, Inc., Sunnyvale, CA, USA) [26] before or had only used one in a brief demonstration setting. Three participants’ results were not analyzed due to one being unable to learn how to teleoperate the robot smoothly, one pulling on the sample too hard and permanently damaging it, and one who did not wear their headphones properly during the training block.

The experiment was a mixed design, with one between-subjects factor and two within-subjects factors. The between-subjects factor was the haptic training condition: Either manual manipulation (M), teleoperation with haptics (TH), or teleoperation without haptics (T). Each participant was assigned to one of the conditions and learned to perform a material tensioning task over a set of reference force levels (Fig. 1). They were then evaluated in a block consisting of two sub-experiments. In the first sub-experiment, participants performed the same tensioning task under the T condition over a different set of reference force levels. In the second sub-experiment, participants performed a palpation task under the T condition over a separate set of reference force levels. The within-subjects factors were the blocks of pseudo-randomized trials and the set of reference force levels the participants were asked to exert. The range of reference force levels for both the training and evaluation blocks were within the 0.5-12 N reported for maneuvers such as cutting, suturing, dissecting, and grasping in minimally invasive surgery [27].

Performance metrics were computed for all valid trajectories. A trajectory was considered valid if the pulling or palpating motion did not contain a false start or more than two reversals. The latter indicates that the participant was not attempting to make a single guess but was probing the system for more information.

In this study, the performance of each participant was quantified by accuracy and speed. Accuracy was measured as force error. This was computed by taking the difference between the force applied to the sample when the user confirmed their guess at the end of their pull, and the target reference force for that attempt. Thus, a positive force error would correspond to overestimation, and vice versa, with values closer to zero implying better accuracy.

Speed was measured using the peak velocity and movement time. In the tension task, the movement onset was computed by identifying the time when velocity first crossed 25% of the peak initial velocity (Fig. 5a). The movement offset was found by finding the last time that the velocity passed 25% of the peak final velocity. The difference between the movement offset and onset times then defined the movement time. Because the end cap that the participant pulled on was attached directly to the sample endpoint, the position information from the MicronTracker was usable in this task. For the palpation task, the probe was not attached to the sample, and thus the amount of displacement of the sample by the probe tip did not correspond with the position data from the MicronTracker. Thus, in order to compute the movement time, the force profile was used instead of the displacement profile, and peak load rates were used instead of peak velocity (Fig. 5b).

As shown by Fitts, slower movements are more accurate [31]. The speed-accuracy tradeoff can be quantified as a function of movement time and error [32][33][34]. A better speed-accuracy function implies that better accuracy can be achieved at equal or faster task performance speed relative to another speed-accuracy function. We chose here to account for the difference in units of accuracy and time by defining a normalized metric, normalized absolute error-time (NAET), as a product of movement time and absolute force error, such that

where $\text{error}_{\text{max}}$ and $\text{time}_{\text{max}}$ are the observed maxima for each quantity during evaluation over all subjects (i.e., all three training conditions). The absolute value of force error is used because taking the product of force error with time reduces the interpretability of the direction component of the metric. For instance, two trials with similar movement times and force error magnitudes, but with opposing force error signs, should be more similar to each other in terms of speed-accuracy than two trials with different movement times and force errors that are similar in both magnitude and sign. The normalization of error and time over their population maximums give unitless quantities between 0 and 1 that assign equal weight to error and time. A lower NAET implies that a participant makes less trade-off between speed and accuracy compared to another participant with higher NAET.

A linear mixed-effects model was fit for each performance metric. The fixed effects were haptic training condition, reference force, experiment progression (trial number), and all their interactions. The random effect was subject modeled as a random intercept. Metrics that were positive valued were log-transformed to fit the assumptions for normality required by the model. An F-test using Satterthwaite’s approximation for the denominator degrees of freedom was performed for each metric to test for significance. For each fixed effect, a p-value of less than 0.05 was deemed as significant. If a fixed effect or an interaction with the haptic training condition was found to be significant, a post-hoc t-test was performed using the Bonferroni correction to test for pairwise differences between conditions.

A stiffness model of force as a function of displacement was fit with a polynomial expression up to the 3rd degree. A nonlinear regression was performed, with the mean final displacement regressed over reference force level in the tension task during evaluation for each haptic training condition. The polynomial components were tested for significance using an F-test. A component was deemed to have a significant effect on the model if its corresponding p-value was less than 0.05.

The learning curves for each haptic training condition are summarized as plots of the mean absolute force error against trial number for each reference force level (Fig. 6). The results indicate that there was a decrease in mean force error over the first 10 trials for all haptic training conditions. Beyond the 10th trial, the mean absolute force error for each condition and force level became more consistent. This suggests that participants had learned to replicate their target reference forces. Towards the end of each set of 30 trials, there was a noticeable increase in force error for several consecutive trials. The increase in force error was more pronounced in the T and TH conditions, for the 4.5 N and 5.5 N force levels. This could have resulted from greater mental fatigue due to participants performing teleoperation, which might be more cognitively demanding than manual operation.

The polynomial models that were fit to the mean displacement over each reference force level had statistically significant quadratic terms for the M and TH conditions and a statistically significant cubic term for the T condition (Fig. 8). For the latter, the fit indicates an initially quadratic relationship of force as a function of displacement that becomes approximately linear as reference force level increases. The shapes of the curves also highlight the observation from Fig. 7a that there is initial overestimation and later underestimation for all haptic training conditions as the force level increases.

The similarities in force error (Fig. 7a) and NAET (Fig. 7d) between all conditions below 5.5 N suggest that when movements and forces are similar to those that were previously learned, there is no use of a haptically informed stiffness model. Instead, the consistent trend across all groups, of underestimation at low reference force levels and overestimation at high reference force levels, supports the use of a displacement-force heuristic.

One explanation is human proprioception bias. Fuentes and Bastian have shown that there is a bias in the estimate of one’s arm position towards more extreme positions such that, when the elbow is extended to an obtuse angle, it is perceived to be at a more obtuse angle than it actually is [35]. Likewise, if the elbow is at an acute angle, it is perceived to be at a more acute angle than it actually is. At high force levels and corresponding high displacements, this belief that the arm is more extended than it really is leads to systematic overestimation of imparted sample displacement and results in force undershoot. At low force levels and low displacements, the effect is the opposite, and the belief that the arm is less extended than it really is leads to systematic underestimation of imparted sample displacement and results in force overshoot.

Another possible explanation for the observed trend is that the movements made during evaluation are biased towards the mean of the movements learned during training. This has been documented in reaching experiments where people experienced linearly increasing forces, but planned movements based on the average of the previously experiences forces as opposed to predicting the increase [36]. Under such assumptions, the observed force error would be smallest at the mean reference force of 3.5 N experienced during training. In Fig. 7a, we see that the point at which the force error crosses zero is close to 3 N, thus lending evidence to this possibility. While the participants in the M and TH conditions learned to perform the task while experiencing environmental forces, those in the T condition did not. Because the experiments by Mawase and Karniel only investigated the effect of prior experience on movements under changing task dynamics [36], the identical trend seen in those under the T condition remains unaddressed by this possible explanation.

The high reliance on proprioception for displacement judgments is consistent with the significantly greater force undershoot by the M group compared to the T and TH groups at the reference force levels outside the training range (Fig. 7a). During training, the teleoperated group (T and TH) operated under a motion scaling ratio of 2:1, which is identical to that in the evaluation block. Meanwhile, the M group trained under a different motion scaling of 1:1. Because the M group never had to estimate force at the large arm displacements encountered by the T and TH groups in training, participants in the M group might not have compensated for their arm position overestimation bias when encountering force-displacement ranges outside what they were exposed to. The learned adaptation to the different dynamics of teleoperation has been previously documented by Nisky et al. in a study of reaches by novices and experienced surgeons [33].

As discussed by Mawase and Karniel, the effect of a bias toward the mean of past experiences of force might be influenced by the familiarity of altered dynamics of the task [36]. Hence, under more well interpreted dynamics, such as object lifting as opposed to viscous force fields, this effect is eliminated [37]. In our experiment, the M group experienced a change in motion scaling ratio, and a switch from manual manipulation to teleoperation, between training and evaluation phases, which could result in greater unfamiliarity with the altered dynamics, thus leading to a stronger effect of the bias towards the mean at previously unseen levels of force.

The larger force underestimation of the M group, compared to the T and TH groups, occurred at force levels relevant to knot-tying and suturing [27]. Given that a proprioception-force heuristic might have be used to perform the tensioning task, underestimation of forces could mean that poor adaptation to task dynamics, like that seen in the M group, could lead to poor performance of the tension-critical portions of suturing and knot-tying maneuvers in RMIS.

The required movement direction in the tension task is horizontal, which is different from the vertical direction of movement required in the palpation task. This makes it unlikely that the same failure to adapt to proprioceptive bias seen in the tension task would have contributed to a consistent bias towards lower force estimates, and a resultant lower NAET for the M group compared to the teleoperated groups. Likewise, there was no prior experience of the dynamics of the palpation task, and hence the effect of any bias towards a mean over prior experience would be mitigated. Instead, the small separation of NAET between the M and TH groups at low force levels (Fig. 7g) provides some evidence that participants belonging to M group were able to leverage past interactions with viscoelastic materials in compression during daily living, to improve force estimation performance.

Through probabilistic mechanisms described by Di Luca and Ernst [38], the sample stiffness experienced by the participants over the training phase would have been integrated with an estimate of stiffness derived from viscoelastic objects they previously experienced over the course of their daily life. These prior models of stiffness would naturally contain a compression component in addition to a tension component. Since stiffness is primarily a relationship between force and displacement, the stiffness model would be correct only if the scale of interaction was similar to that of daily living. This can only occur in the M training condition, and thus the access of such a haptically informed stiffness model offers an explanation for the lower NAET of the M group compared to the TH group. Because force information was still available through color change, the better performance of the T group compared to the TH group in NAET is possibly explained by the former having developed better accuracy in interpreting the color change as a force cue during training.

The better speed-accuracy function of the M group compared to the TH group occurred at force levels relevant for palpation in minimally invasive surgery [27]. However, they are below the average forces reported for dissection, another maneuver that can require compression forces [16][27]. Thus, the benefits of the better speed-accuracy function seen for the M group might imply better performance in RMIS only for palpation.

While force estimation performance of participants during evaluation were differentiated by the task dynamics experienced in training, the resultant estimated stiffness models they formed for the tension task in evaluation were instead differentiated by the availability of haptic feedback in training. The shape of the estimated force-displacement curve for the T condition in Fig. 8 suggests that participants were performing a more linear extrapolation for the forces outside the training range compared to the M and TH groups. Thus, while no group correctly estimated the non-linear stiffness of the silicone throughout the entirety of the tested force range, the haptics groups did capture the upward concavity of the force-displacement curve that was characteristic of the sample at the higher force range. This was reflected analytically in the cubic terms of the regression fits for the M and TH groups, which were not found to be significant, unlike that of the T group. This is consistent with the results of Wu and Klatzy, who provide evidence of a model of stiffness that recursively updates throughout an interaction [39]. They also found that the final estimate of stiffness is highly weighted towards that which was experienced towards the end of the interaction. In our experiment, the haptic training conditions (M and TH) provided more continuous force information compared to the post-trial error feedback available to the T group. This would have allowed the participants in the haptic training conditions to capture more of the non-linearities of the sample stiffness and consequently weight the upward concave stiffness experienced during the latter part of the interaction more highly, thus arriving at the quadratic stiffness estimate.

In this experiment, participants were not given instructions to make fast movements, which could have contributed to failure to learn a prior of stiffness during the tensioning task. Instructions that shift attention away from internal movement control contribute to movement automatization [40][41], or implicit learning [42], which promotes better retention and generalization [43]. Due to better cognitive efficiency, implicitly learned movements result in better performance, and are thus rewarded more under time constraints, compared to explicitly learned movements [44]. Hence, the provision of an instruction to make fast movements could have promoted implicit learning of a cognitively efficient stiffness model as opposed to an explicit heuristics-based strategy to perform the tasks. Future work should investigate if the promotion of fast movements during the training phase can encourage the development and use of stiffness models to improve performance and generalization of force estimation in the learned task.

While the results of both evaluation tasks suggest that the amount of motion scaling during training is likely to have an effect on force estimation ability, only one ratio of scaling was investigated in this study. To further investigate the effect of scaling on force estimation in RMIS, future experiments should train participants at different ratios of motion scaling.

Because of its simplicity and short timescale, the results of this study might not be fully relevant to the experience of clinical RMIS training, which is known to involve tasks of greater complexity and duration. The tension task is simple compared to those in other haptics-related RMIS studies like blunt dissection, which requires making a variety of scraping movements in multiple directions [16], needle driving, which requires grasping a needle and driving it along a curved path through soft material [45], and even peg transfer, which involves grasping and lifting rubber samples from a peg with minimal damage and high positional accuracy [17]. The above tasks engage implicit mechanisms of force estimation to achieve higher-order goals, and are similar to the critical points at which attendings gauge trainees’ force sensitivity in rubrics such as GEARs. Our simple task on the other hand, requires participants to reason directly in terms of forces and is likely to engage more explicit mechanisms of force estimation. Thus, participants might not have been required to develop a prior representation of stiffness of the material to achieve satisfactory performance, instead successfully relying on a heuristic mapping of force to displacement. While there has been evidence to show that explicit and implicit mechanisms can be developed in parallel even for simple tasks such as point-to-point reaching [46], it is also possible that the two are independent [47]. Thus, it is unclear if our work can generalize to more complex tasks like those found in RMIS. The short duration of the study also might not have afforded enough time for participants to learn a prior representation of stiffness, as it typically takes expert surgeons a long period of time to develop good tissue-handling skills with a surgical robot. This slower rate of implicit learning has been documented in prior work [46]. Furthermore, there is evidence suggesting that skill retention is less for explicit learning compared to implicit learning [48].

Our experiment sought to mitigate these limitations by focusing on tasks with simple dynamic models, learned under conditions that reduce reliance on external cues. Our findings suggest that prior haptic experience potentially helps with the generalization of force estimation to simple, unseen tasks. Future work is needed to build upon our basic result, and investigate if haptically informed models of stiffness can be learned in more complex and realistic tasks, likely over longer periods of time. Additionally, the use of a more complex, clinically relevant task would facilitate baseline comparisons with expert surgeons, and thus offer greater insight into how meaningful any effect of haptic training would be in developing force estimation skill in RMIS.