Haptics-Enabled Forceps with Multi-Modal Force Sensing: Towards Task-Autonomous Surgery

The design of the sensing module (Fig. 1(e)) has followed two requirements: 1) small in size, 2) adaptive to micro-sized forceps, for example, the biopsy forceps. The module is designed in a cylindrical shape to minimize anisotropy, and the main components are a flexure, a camera, and a target. A central channel is reserved for the passage of instruments. The target, supported by the flexure, is manufactured with multiple holes that allow light to pass from a light source, and these holes can then be captured by a camera as markers for the estimation of the target’s pose, as depicted in Fig. 1(e). The module’s size and stiffness can be customized based on the instruments’ and applications’ needs. Here, we assemble a prototype with a diameter of 4mm and a length of 12mm and integrate it in a pair of biopsy forceps, as shown in the lower-right of Fig. 1(e).

Material properties and geometry play a dominant role in the multidimensional stiffness of the flexure. Different approaches have been adopted for flexure design, including the pseudo-rigid-body replacement method [26], topology optimization [27], etc. For force sensing in different surgical tissue manipulations, the flexure should also be changeable according to the task requirements. One commercial and readily available flexure is the compression spring [21, 22]. Although compression springs typically show dominant stiffness in the axial direction, they also exhibit a certain degree of stiffness in lateral directions and can be linearly approximated [28]. Moreover, they can be easily reconfigured by choosing different wire diameters, outer diameters, and coil numbers. For the prototype, we chose a stainless steel compression spring with a wire diameter of 0.5mm, outer diameter of 4mm, rest length of 5mm, and four coils. The selected spring has a force response range of 0$\sim$5N and 0$\sim$2N at axial and lateral displacements of 0$\sim$2mm and 0$\sim$1mm, respectively, which can meet the requirement of most tissue manipulations [29].

The camera for the sensing module should have a compact size, sufficient resolution, and surgery compatibility. For the prototype, we chose OVM6946 (Omnivision Inc., USA), a medical wafer-level endoscopy camera, which has been adopted in many surgical applications [30, 5]. It is 1mm in width and 2.27mm in length and has a 400$\times$400 resolution and 30Hz rate. To ensure continuous tracking and estimation, we fabricated the target with 12 cycle-distributed 0.15mm-diameter holes used as markers for the camera to track and estimate the target pose, and a central hole is reserved for tools to pass through. These holes can also be customized based on the instruments and applications. Considering the integration into the biopsy forceps for MIS, the prototype’s target in Fig. 1(e) has a 3.4mm outer diameter and a 0.6mm-diameter central hole for the driving cable.

In order to estimate the target pose, each marker needs to be tracked by the camera first. However, as the raw image captured by the camera is full of noise, as seen in Fig. 2(b), markers cannot be detected robustly. Therefore, a bilateral filter and threshold are adopted to minimize the noise and preserve the boundary in images [31]. Then, markers are circled by a parameterized blob detection [32], as shown in Fig. 2(c). Finally, each marker’s central pixel position ${}^{I}{\mathbf{p}}$ is returned.

The positions of markers ${}^{M}{\mathbf{P}}\in\mathbb{R}^{2\times 12}$ in the target frame {M} are known. The projection between the $i$-th marker centre ${}^{M}{\mathbf{p}}_{i}=[x_{i},\,y_{i}]^{\textnormal{T}}$ and its corresponding pixel ${}^{I}{\mathbf{p}}_{i}=[u_{i},\,v_{i}]^{\textnormal{T}}$ in the image frame {I}, as shown in Fig. 2(a), can be formulated as

where ${}_{M}^{C}{\mathbf{T}}=\left[\begin{array}[]{*{20}{c}}\mathbf{r}_{1}&\mathbf{r}_{2}&\mathbf{r}_{3}&\mathbf{t}\\ 0&0&0&1\end{array}\right]=\left[{\begin{array}[]{*{20}{c}}r_{11}&r_{12}&r_{13}&t_{x}\\ r_{21}&r_{22}&r_{23}&t_{y}\\ r_{31}&r_{32}&r_{33}&t_{z}\\ 0&0&0&1\end{array}}\right]$ is the transformation from camera frame {C} to marker frame {M}, ${}^{M}\tilde{\mathbf{p}}_{i}=[x_{i},\,y_{i},\,0,\,1]^{\textnormal{T}}$ is the homogeneous form of ${}^{M}{\mathbf{p}}_{i}$, ${}^{I}\tilde{\mathbf{p}}_{i}=[u_{i},\,v_{i},\,1]^{\textnormal{T}}$ is the homogeneous form of ${}^{I}{\mathbf{p}}_{i}$, $i\in[0,n]$ ($n=11$ for the prototype), $f$ is the camera’s focal length, $\rho_{w}$ and $\rho_{h}$ are the width and height of each pixel, respectively, $z={(h_{31}x_{i}+h_{32}y_{i}+1)}\,t_{z}$, and the camera intrinsic matrix $\mathbf{A}$ is obtained via calibration with MATLAB camera toolbox [33].

According to the transformation relationship shown in Fig. 2(a), the current target pose related to its initial state ${}^{M_{0}}_{M_{t}}\mathbf{T}$ can be formulated as

The translation of ${}^{M_{0}}_{M_{t}}\mathbf{T}$ is equal to the displacement $\mathbf{d}_{s}$ in (1). Then, the question becomes to get ${}^{C}_{M_{0}}\mathbf{T}$ and ${}^{C}_{M_{t}}\mathbf{T}$. The relationship between ${}^{C}_{M}\mathbf{T}$ and homography matrix $\mathbf{H}$ can be formulated as

Because $\mathbf{r}_{1}$ is a unit vector, $t_{z}$ can be calculated by $t_{z}=\frac{1}{||\mathbf{A}^{-1}[h_{11},h_{21},h_{31}]^{\rm T}||}$. Therefore, to estimate ${}^{C}_{M_{0}}\mathbf{T}$ and ${}^{C}_{M_{t}}\mathbf{T}$, we only need to solve the homography matrix $\mathbf{H}^{0}$ and $\mathbf{H}^{t}$ of the initial and current states.

Because each homography matrix $\mathbf{H}$ has eight variables, at least four independent projections between ${}^{M}{\mathbf{p}}_{i}$ and ${}^{I}{\mathbf{p}}_{i}$ are needed to solve it, and each pair’s relationship can be formulated as

where $({}^{M}{p_{i,x}},{}^{M}{p_{i,y}})$ and $({}^{I}{p_{i,u}},{}^{I}{p_{i,v}})$ are coordinates of the $i$-th marker in the frame {M} and its projection in the frame {I}. Based on this relationship and four pairs of projections, a set of equations can be established to solve $\mathbf{H}$. Substituting the solved $\mathbf{H}$ into (31), ${}^{C}_{M}\mathbf{T}$ can be further obtained.

Ideally, through substituting four detected ${}^{I}{\mathbf{p}}_{i}$ and their corresponding ${}^{M}{\mathbf{p}}_{i}$ into (32) we can calculate the homography matrix $\mathbf{H}$ for each state. However, under external forces, the pose variation of the target causes the occlusion or halation of some markers and results in wrong correspondence between the image and the marker. As a result, the corresponding relationship between ${}^{I}{\mathbf{p}}_{i}$ and ${}^{M}{\mathbf{p}}_{i}$ changes over time, which invalidates the direct use of (32). Therefore, a registration process is required to label the currently detected markers’ pixel positions ${}^{I}{\mathbf{P}}^{t}$ to their original indexes, as shown in Fig. 2(a), and we define this registered result as ${}^{I}{\mathbf{P}}^{t,r}$.

At the initial state $t=0$, the target is installed perpendicularly to the optical axis. For the prototype we built, all the markers can be detected except the 11^th marker (top marker) that is occluded by the channel, as shown in Fig. 2(b). Therefore, we define ${}^{M}{\mathbf{P}^{0}}$ by removing the 11^th marker from ${}^{M}{\mathbf{P}}$ as the marker set at the initial state, and the projection $\mathbf{H}^{0}$ between ${}^{M}{\mathbf{P}^{0}}$ and ${}^{I}{\mathbf{P}}^{0}$ can be obtained according to (32). Substituting ${}^{M}{\mathbf{p}}_{11}$ and $\mathbf{H}^{0}$ back to (32), we can estimate ${}^{I}{\mathbf{p}}^{0}_{11}$. Then, the complete pixel-position array of the initial state ${}^{I}{\mathbf{P}}^{0,c}$ can be obtained by appending ${}^{I}{\mathbf{p}}^{0}_{11}$ to ${}^{I}{\mathbf{P}}^{0}$.

For the current state $t$, the pixel positions of the detected markers ${}^{I}{\mathbf{P}}^{t}\in\mathbb{R}^{2\times m}$ ($m$ is the number of detected markers, $m\leqslant n$) are returned by the blob detection. Nevertheless, these detected markers are directly labelled from bottom to top in the image. For example, the blob detection result of a random pose is shown in Fig. 2(d). Assuming ${}^{I}{\mathbf{P}}^{t-1,c}$ is known, we calculate the L1 norm of the distance vector between each ${}^{I}{\mathbf{p}}^{t}_{i}$ and ${}^{I}{\mathbf{p}}^{t-1,c}_{j}$, $i\in[1,m]$ and $j\in[1,n]$, and define it as $e$. The corresponding marker of ${}^{I}{\mathbf{p}}^{t}_{i}$ in ${}^{I}{\mathbf{P}}^{t-1,c}$ is the one that results in the minimal $e$, and its index is the desired $j^{*}$. Without loss of generality, the registration problem for the detected i-th hole is defined as

Repeating this registration for each detected marker ${}^{I}{\mathbf{p}}^{t}_{i}$ and updating ${}^{I}{\mathbf{p}}^{t,r}_{j^{*}}\leftarrow{}^{I}{\mathbf{p}}^{t}_{i}$, we can obtain ${}^{I}{\mathbf{P}}^{t,r}$ with the original indexes. We define the corresponding marker position set of ${}^{I}{\mathbf{P}}^{t,r}$ as ${}^{M}{\mathbf{P}}^{t}$. Substituting ${}^{I}{\mathbf{P}}^{t,r}$ and ${}^{M}{\mathbf{P}}^{t}$ to (32), the homography matrix of the current state $\mathbf{H}^{t}$ can be calculated.

To ensure the image of the current state can be used as the reference for the next registration, i.e., the $t$+1 state, the complete pixel-position set ${}^{I}{\mathbf{P}}^{t,c}$ of the current state also should be generated. By substituting undetected markers’ coordinates in ${}^{M}{\mathbf{P}}$ and $\mathbf{H}^{t}$ to (32), we can estimate their corresponding pixel positions ${}^{I}{\mathbf{P}}^{t,e}$. Then, ${}^{I}{\mathbf{P}}^{t,c}$ can be obtained by merging ${}^{I}{\mathbf{P}}^{t,e}$ with ${}^{I}{\mathbf{P}}^{t,r}$. For example, Fig. 2(e) shows the registration result and the generated complete image of Fig. 2(d).

The transformation between the target and the camera at the initial state ${}^{C}_{M_{0}}\mathbf{T}$ can be calculated by substituting ${}^{I}{\mathbf{P}}^{0}$ and $\mathbf{H}^{0}$ into (31). Similarly, the transformation at the current state ${}^{C}_{M_{t}}\mathbf{T}$ can be obtained by substituting ${}^{I}{\mathbf{P}}^{t,r}$ and $\mathbf{H}^{t}$ into (31). Then, the transformation ${}^{M_{0}}_{M_{t}}\mathbf{T}$ can be calculated by substituting ${}^{C}_{M_{0}}\mathbf{T}$ and ${}^{C}_{M_{t}}\mathbf{T}$ into (26), which reflects the deformation of the flexure.

The displacement of the spring $\mathbf{d}_{s}$ is the translation vector of ${}^{M_{0}}_{M_{t}}\mathbf{T}$. Substituting $\mathbf{d}_{s}$ and the spring’s stiffness matrix $\mathbf{K}_{s}$ into (1), we can estimate the external force $\mathbf{f}_{s}$. Ideally, the stiffness matrix $\mathbf{K}_{s}$ represents the spring’s characteristics that connect the camera and the target. However, this stiffness matrix $\mathbf{K}_{s}$ should also consider the influence of wires and shelter, as they are parallel to the spring. In order to get the practical stiffness matrix $\mathbf{K}_{s}$, a calibration is carried out on the prototype and detailed in section III. In summary, the above pose and force estimation method can be summarized as algorithm 1.

To drive the forceps and conduct experimental verification, we designed a micro-level actuator, as shown in Fig. 1(d), which consists of upper and lower linear drivers that are responsible for grasping and pushing/pulling, respectively. The lower driver also compensates for the motion introduced by the flexure’s deformation when the upper driver actuates the forceps to grasp tissue. The compensation is achieved by moving the lower driver’s carrier for the same amount of the flexure’s axial deformation estimated by the camera in the reverse direction. A commercial single-axis force sensor (ZNLBS-5kg, CHINO, China), with a resolution of 0.015N, is installed collinearly to the forceps and connected to the driving cable to measure the driving force.

Integrating the sensing module into the forceps, we can enable its haptic sensing, and the haptics-enabled forceps are shown in Fig. 1(e). The base of the forceps is installed concentrically to the sensing module’s head, and the driving cable passes through the reserved central channel. The cylindrical base connects the forceps to its carrying instrument (stainless tube). A prototype is also shown in the inset of Fig. 1(e), which has a 4mm diameter and 22mm total length and is used in the following description and experiments. The proposed haptics-enabled forceps can be easily integrated into a robotic surgical system with the micro-level actuator, such as the robot presented in Fig. 1(a). The forces applied to the forceps when they push or pull a tissue are also shown in Fig. 1(e). $\mathbf{f}_{p}$ is the pushing/pulling force from the interactive tissue. $\mathbf{f}_{s}$ is the elastic force from the spring, which can be directly estimated by (1). $\mathbf{f}^{\prime}_{d}$ is the driving force of the cable at the tool end. Ignoring friction, we assume $|\mathbf{f}^{\prime}_{d}|$ equals $|\mathbf{f}_{d}|$ measured by the single-axis force sensor connected to the driving cable at the proximal side. Then, $\mathbf{f}^{\prime}_{d}$ can be formulated as $\mathbf{f}^{\prime}_{d}={}^{M_{0}}_{M_{t}}{}\mathbf{T}\,{}\mathbf{f}_{d}$. Finally, we can obtain the pushing/pulling force $\mathbf{f}_{p}$ as

For tissue touching, the contact force can also be estimated by (34) no matter if the forceps are closed or open.

The estimation of the grasping force is more challenging because it relates to the geometry and driving force applied to the forceps’ cable, as depicted in Fig. 3. Although the sensing module can estimate the 3D elastic force $\mathbf{f}_{s}$ and the orientation of $\mathbf{f}^{\prime}_{d}$, we mainly consider the situation where the forceps grasp and pull tissue straightly as grasping is performed in the release condition.

The forces applied to the forceps when they grasp and pull a tissue are shown in Fig. 3(c). For convenience, we adopt $F_{-}$ to denote the magnitude of force $\mathbf{f}_{-}$ in the following sections. $F_{s}$ is the elastic force from the spring and is directly estimated by (1). $F_{d}$ is the driving force applied to the forceps’ hinge and measured by the proximal single-axis force sensor. $F_{g}$ and $F_{f}$ are contact and friction forces from the grasped tissue, as shown in Fig. 3(c). For ease of understanding, we define $F_{g}$ as the gripping force in the following description.

Assuming the momentum of one jaw about joint $j_{2}$ is zero, as sketched in the right inset of Fig. 3(c), we can further obtain the relationship between $F_{g_{1}}$ and $F_{d}$ as

where $\alpha=\frac{\theta}{2}+\alpha_{0}$, the definitions of $\theta$, $\alpha$, $\alpha_{0}$, $l_{2}$, and $l_{3}$ are shown in Fig. 3(b). $\theta$ and $\alpha_{0}$ can be calculated based on the geometric relationship of the forceps’ links. Fig. 3(a) shows the forceps’ geometry at the initial state, i.e., the forceps are in the closed form, and the flexure is relaxed. According to the relationship of forceps’ links at the initial state, we can get $\alpha_{0}$ and formulate it as

where $l_{1,2}$ is the distance between forceps’ two joints $j_{1}$ and $j_{2}$. When the forceps are open at angle $\theta$, the geometry changes to Fig. 3(b). At this stage, the angle ${\alpha}$ can be reformulated as

where $l{{}_{1,2}^{\prime}}=l{{}_{1,2}}+{t}_{d}-{t}_{s}$, $t_{d}$ is the movement of the driving cable measured by the upper driver’s encoder of the micro-level actuator, and $t_{s}$ is the transformation of the sensor’s head estimated by the sensing module. Moreover, the relationship of $\theta$, $\alpha$, and $\alpha_{0}$ is

For a pair of forceps, we assume $F_{g_{1}}$ equals $F_{g2}$, and $F_{f_{1}}$ equals $F_{f_{2}}$. Then, referring to (34), we can formulate $F_{g}$ as

Fig. 4(a) shows the experimental setup for pose estimation evaluation with an electromagnetic (EM) tracking system (Northern Digital Inc, Canada) with resolutions less than 0.1mm in position and 0.1^∘ in orientation. The EM tracking sensor was installed concentrically to the proposed force sensing module. During the experiment, the sensor’s head was moved along $x$ (red arrow), $y$ (green arrow), and $z$ (blue arrow) as indicated in Fig. 4(b).

The orientation comparison is based on nine pairs of samples distributed in the sensing module’s workspace, as plotted in Fig. 4(c). Here, we use ($\alpha_{C,i}$, $\beta_{C,i}$, $\gamma_{C,i}$) and ($\alpha_{E,i}$, $\beta_{E,i}$, $\gamma_{E,i}$) to denote the pitch, roll, and yaw angles of camera estimation and EM tracking results of the $i$-th pair. The max mean and absolute maximum deviations between EM tracking and our estimation result are calculated by $\max\frac{1}{n}({\sum\limits_{i=1}^{n}{{\alpha_{C,i}}-{\alpha_{E,i}}}}\,,{\sum\limits_{i=1}^{n}{{\beta_{C,i}}-{\beta_{E,i}}}}\,,{\sum\limits_{i=1}^{n}{{\gamma_{C,i}}-{\gamma_{E,i}}}})$ and $\max(\max\limits_{i=1}^{n}({\alpha_{C,i}}-{\alpha_{E,i}})\,,\max\limits_{i=1}^{n}({\beta_{C,i}}-{\beta_{E,i}})\,,\max\limits_{i=1}^{n}({\gamma_{C,i}}-{\gamma_{E,i}}))$, where $n=9$. For position evaluation, the EM sensor and target were tracked continuously by the EM tracking system and camera, respectively, and the comparison is shown in Fig. 4(d). To calculate the mean and max deviations between the EM and camera-tracking traces, we further calculated the mean ($d_{a}$) and Hausdorff ($d_{h}$) distance by

where ${d_{{C_{i}},E}}=\mathop{\min}\limits_{j=1}^{m}(||{\mathbf{p}_{C,i}-\mathbf{p}_{E,j}}||_{2})$, $\mathbf{p}_{C,i}$ and $\mathbf{p}_{E,j}$ denote positions of the camera estimation of point $i$ and EM tracking of point $j$, ${d_{{E_{j}},C}}=\mathop{\min}\limits_{i=1}^{n}(||{\mathbf{p}_{E,j}-\mathbf{p}_{C,i}}||_{2})$, $n$=1336 and $m$=2181 are the numbers of points recorded by the camera and EM tracking system, respectively. As listed in Table I, the max mean, absolute maximum, and root mean square (RMS) deviations of the orientation are 0.056rad, 0.147rad, and 0.041rad, and those of the position are 0.0368mm, 0.2265mm, and 0.0551mm. This indicates that the proposed method can track and estimate the target’s pose reliably.

We used a series of weights to calibrate the haptics-enabled forceps’ stiffness matrix, considering the influence of the driving cable and soft shell. The calibration platform is shown in Fig. 5(a). Here, the micro-level actuator is used to adjust the position and hold the forceps. An orientation module is installed concentrically to the forceps for adjusting the direction of the pulling force provided by a cable. This cable is connected to the forceps’ jaws, and the force $F_{w}$ applied to it is adjusted by adding/removing weights.

Two groups of data were collected for calibration and verification, respectively. When collecting the data for calibration in the $x$-$y$ panel, as shown in Fig. 5(b), the orientation module made the cable align with the $x$ and $y$ axis, and the force was increased from 0N to 0.49N at intervals of 0.07N. Then, for the $z$ axis, the driving force was increased from 0N to 5N at intervals of 0.5N. The target pose for each interval was recorded. Then, substituting the calibration data to $\mathbf{K}_{s}=\mathbf{f}_{s}/\mathbf{d}_{s}$ in MATLAB, we obtained the stiffness matrix $\mathbf{K}_{s}$ as

To verify the calibrated $\mathbf{K}_{s}$, we rotated the orientation module for $45^{\circ}$, as shown in Fig. 5(c), and collected the data following the same procedure of calibration. For the $x$-$y$ panel the interval was set to 0.035N, while for the $z$ axis the interval was 0.02N. The comparison of weights and estimated forces are plotted in Fig. 5(d) and (e), which show the results of the $x$-$y$ panel and the $z$-axial direction, respectively. The absolute mean, maximum, and RMS errors of ($f_{x}$, $f_{y}$, $f_{z}$) in these three comparisons are ($0.0064$, $0.0045$, $0.0383$)N, ($0.0233$, $0.0154$, $0.1996$)N, and (0.8494, 0.5884, 4.9367)N, which are (1.86%, 1.30%, 0.76%), (6.71%, 4.46%, 3.97%), and (2.45%, 1.70%, 0.98%) of the measured force amplitude (MFA), respectively. This indicates that the calibrated $\mathbf{K}_{s}$ can be used to estimate the force applied to the forceps.

Fig. 7(a) shows the experimental setup, where a commercial pressure sensor (FlexiForce A201-1 lbs, Tekscan, USA), with a resolution of 0.02N, and a force sensor (ZNLBS-5kg, CHINO, China) were adopted as references. The pressure sensor was grasped by the forceps for direct measurement of the grasping force. Because the pressure sensor’s sensing area was a 10mm-diameter circle, 3D-printed additional surfaces were attached to the forceps’ jaws to ensure sufficient contact. The driving force reached around 7N during the grasping action in each experiment. Since, from (39), we can see the grasping force $F_{g}$ is also related to pulling force $F_{p}$, we connected the pressure sensor to a commercial force sensor by a flexure to measure the applied pulling force.

We carried out three experiments with different $\theta$ configurations. Each experiment was presented as a continuous process, and the results are shown in Fig. 7(b), (c), and (d), where the subscript $i\in$(1,2,3) denotes the $i$-th experiment that corresponds to $\theta=10^{\circ}$, $30^{\circ}$, and $50^{\circ}$. The forceps grasped the pressure sensor and reached the target grasping force during 0$\sim$18s. After a pause phase (18$\sim$26s), the forceps pulled the grasped sensor backward for 25mm in the pulling phase (26$\sim$65s).

Fig. 7(b) depicts $F_{d}$ provided by the driving cable and $F_{s}$ measured by our proposed sensing module. $F_{s}$ and $F_{d}$ were almost equal in the grasping and pause phases. This is because the pulling force applied to the forceps was almost zero in these two phases, as there was no pulling action, and the pressure sensor was floating. On the contrary, in the pulling phase, $F_{d}$ increased while $F_{s}$ kept constant. This is because the pressure sensor has been pulled ($F_{p}=F_{d}-F_{s}$), but the flexure deformation kept constant. The difference between $F_{d}$ and $F_{s}$ was the pulling force $F_{p}$ applied to the pressure sensor. As the three experiments followed the same procedure, their data almost overlay each other. However, because $\theta$ is configured differently, the time for performing the grasping is slightly different. The inset in Fig. 7(b) shows that the bigger $\theta$ is, the quicker the grasping finishes.

Fig. 7(c) shows the commercial sensor measured pulling force $F_{p_{r,i}}$ in the experimental procedure, and we compared it with forceps estimated result $F_{p_{e,i}}$. We can see that the pulling force of these three experiments almost overlay each other as the pulling distances were equal. A comparison between the commercial sensor results $F_{p_{r}}$ and our estimations $F_{p_{e}}$ is listed in Table III. The mean, maximum, and RMS errors of the three experiments are under 0.133N, 0.497N, and 0.167N, respectively. This also reflects that the forceps are valid for estimating the pulling force when they grasp tissue.

Fig. 7(d) compares the grasping force $F_{g}$, where $F_{g_{e,i}}$ and $F_{g_{r,i}}$ denote the forces of $i$-th experiment estimated by our proposed forceps and commercial pressure sensor, respectively. Because of the additional surfaces that are shown in the inlet of Fig. 7(a), $l_{3}$ in (39) was set as $l^{\prime}_{3}$ in these experiments. According to Fig. 7(d), we can see that when the driving force $F_{d}$ is equal, the grasping force $F_{g}$ is positively related to $\theta$. This phenomenon conforms with the force calculation method (34) and (39), as the increased $F_{p}$ could entail greater $F_{d}$ and then result in increased $F_{g}$. Table III lists the absolute mean, maximum, and RMS errors, and their percentage of MFA, between $F_{g_{r}}$ and $F_{g_{e}}$ of the three experiments. Their values are under 0.040N, 0.133N, and 0.049N, respectively. This indicates that the forceps are valid for estimating the grasping force and meet the requirements of surgical applications [29].

To further verify the feasibility of the proposed system, we conducted a group of robotic experiments on an ex vivo tissue. Fig. 8 shows the experimental setup, where a UR-5 robotic arm was adopted as the macro-level actuator. The ex vivo chicken tissue was placed in a human body phantom to simulate the lesion, and the instrument was implemented through a simulated minimally invasive port on the phantom.

We carried out automatic tissue grasping procedure with one targeted grasping force $F_{g}^{\ast}$ and two different targeted pulling forces $F_{p}^{\ast,1}$ and $F_{p}^{\ast,2}$. The key experimental scenes and results are shown in Fig. 9, and the experiments can be roughly divided into five phases. Transformations between these phases were automatically performed depending on the grasping force $F_{g}$ and pushing/pulling force $F_{p}$ estimated by the haptics-enabled forceps.

With Mode-I enabled, the forceps were driven to touch the tissue during the touching phase (0$\sim$7.5s) until $F_{p}$ reached the touching detection threshold $F_{p}^{\ast,t}$ followed by a short pending period (7.5$\sim$9.5s). Then, Mode-II was enabled, and the forceps performed the tissue grasping in the grasping phase (9.5$\sim$13.5s) until $F_{g}$ reached the targeted value $F_{g}^{\ast}$. Finally, with another short pending period (13.5$\sim$15.5s), the grasped tissue was pulled up and held for a while with targeted pulling force $F_{p}^{\ast}$. Because the targeted pulling force $F_{p}^{\ast}$ for the first and second group experiments were different, the time consumed for pulling varied. For the first group with $F_{p}^{\ast,1}=0.4$N, the period was around 6s, while for the second group with $F_{p}^{\ast,2}=0.2$N, it was around 2s.

Fig. 9(b) shows the measured driving force $F_{d}$ and the estimated elastic force $F_{s}$ of these experiments. Fig. 9(c) shows the estimated $F_{g}$, where $F_{g}^{\ast}=0.4$N is the targeted grasping force for grasping action. Fig. 9(d) shows the estimated $F_{p}$, where $F_{g}^{\ast,t}=-0.05$N is the threshold for touch detection, while $F_{p}^{\ast,1}=0.4$N and $F_{p}^{\ast,2}=0.2$N are two target pulling forces for the first and second group experiments, respectively. According to the experiment results, we can see that the haptics-enabled forceps can be implemented in robotic surgery for multi-modal force sensing. We also noticed that, during the grasping process, the pulling force $F_{p}$ varies notably, which can potentially indicate successful grasping.