KineDex: Learning Tactile-Informed Visuomotor Policies via Kinesthetic Teaching for Dexterous Manipulation

Most existing methods for collecting demonstrations with dexterous hands rely on retargeting human hand motion via teleoperation [31, 32, 33, 34, 35] or using video data [19, 20, 21, 22, 23]. However, both approaches share a key limitation: the operator lacks real-time tactile feedback, adversely impacting the efficiency and success rate of data collection. To mitigate this, some studies have introduced exoskeleton systems [36, 24, 25] that simulate haptic feedback during interaction. Nevertheless, such approximations differ fundamentally from the actual tactile sensations experienced through direct physical contact.

Kinesthetic teaching [37, 29, 38, 39, 28] enables operators to physically manipulate the robotic hardware and receive real-time force feedback. HIRO [39] introduces a hand-over-hand teaching paradigm, where the demonstrator grasps the robotic hand to experience accurate force feedback, but does not record tactile data, limiting its applicability to complex tasks. The most closely related work to ours is DexForce [28], which collects tactile-augmented demonstrations via kinesthetic teaching but relies on simpler hardware with fewer degrees of freedom and evaluates only on basic tasks such as grasping or flipping. Moreover, due to visual occlusions from the demonstrator’s hand, DexForce requires replaying kinesthetic trajectories to obtain clean observations—a process that becomes infeasible for longer-horizon tasks.

Learning from expert human demonstrations [40, 41, 42, 43, 44, 45] enables embodied agents to acquire autonomous manipulation skills. Diffusion Policy [44], which applies diffusion models to imitation learning, has shown strong capability in capturing the multimodal structure of demonstration data. Extensions such as 3D Diffusion Policy [45] integrate point cloud information into the perception pipeline, allowing for richer environmental representations. Other works have explored the integration of tactile sensing [46, 47, 48], enabling policies to perform more fine-grained manipulation. In our work, we adopt Diffusion Policy as the backbone for policy learning, augmenting its observation space with tactile sensing and applying force control during inference to ensure precision and stability in contact-rich manipulation tasks.

In this work, we aim to train tactile-informed visuomotor policies from kinesthetic demonstrations via imitation learning, and to implement force control at inference time for executing contact-rich manipulation tasks. The observation space and learning targets are defined as follows.

Observation Space. At each time step $t$, the policy receives multi-modal observations comprising: (i) RGB images captured from $N_{o}$ multi-view cameras, denoted as $o_{t}\in\mathbb{R}^{N_{o}\times H\times W\times 3}$; (ii) tactile vectors obtained from $N_{q}$ sensing points on each of the five fingertips, represented as $q_{t}\in\mathbb{R}^{5\times N_{q}}$; (iii) proprioceptive signals from $N_{x}$ joints of the robotic hardware, given by $x_{t}\in\mathbb{R}^{N_{x}}$.

Action Space. To enable precise force control during inference, the policy predicts not only the target joint positions $x_{d}\in\mathbb{R}^{N_{x}}$, but also the $N_{f}$-dimensional target fingertip forces $f_{d}\in\mathbb{R}^{5\times N_{f}}$, resulting in a force-informed action denoted by $a_{t}=\{x_{d},f_{d}\}$. While $x_{d}$ specifies the baseline configuration of the hand, $f_{d}$ modulates the contact forces via a modified PD controller (introduced in Section 3.4), allowing the fingers to actively track the desired contact forces during execution.

The general hardware setup of KineDex consists of a robotic arm equipped with a dexterous hand, as illustrated in Figure 1. In addition, we employ two RGB cameras to capture visual observations: one is mounted in front of the workspace to provide a global view of the scene, and the other is wrist-mounted on the end-effector to enable close-range perception of the manipulation area.

The core idea of KineDex data collection system is to allow operators to “wear” the dexterous hand while moving freely to perform precise contact-rich manipulation in real time. To enable this hand-over-hand control, we attach ring-shaped straps to the dorsal sides of the four non-thumb fingers of the robotic hand, allowing the operator to guide the hand as if wearing a glove. This physical coupling ensures that contact forces experienced during motion are immediately transmitted to the operator’s hand, providing natural haptic feedback throughout the demonstration. Due to morphological differences between human and robotic hands, the operator controls the thumb separately with their left hand, while guiding the remaining fingers with their right hand as described above. Examples of kinesthetic demonstrations are shown in Figure 4.

During kinesthetic teaching, the following data modalities are recorded for each demonstration:

• •

  Visual observations: RGB images captured from the front-facing and wrist-mounted cameras. As raw observations may contain the operator’s body, we apply an inpainting strategy to address this issue, as detailed in Section 3.3.

• •

  Proprioception: The robot arm’s end-effector pose and the dexterous hand’s joint positions.

• •

  Tactile sensing: Per-finger tactile measurements, with each finger equipped with multiple sensing points that record localized contact forces, forming a dense tactile sensing matrix.

• •

  Fingertip force: A 3D force vector $\mathbf{f}=(f_{x},f_{y},f_{z})$ for each fingertip, where each component represents the force along the corresponding axis, computed by aggregating the localized forces from all tactile sensing points.

The raw kinesthetic demonstrations collected by the system cannot be directly used for visuomotor policy learning, as the front-facing camera inevitably captures the operator’s body during interaction. Training on such data introduces a significant out-of-distribution(OOD) shift at inference time, when the human body is no longer present in the scene (as shown in Table 1). Motivated by recent advances in human-to-robot data editing [49, 50], we adopt an inpainting-based approach to remove the operator’s body from the visual observations.

For raw kinesthetic demonstrations, we first apply Grounded-SAM [51] to extract masks of the operator’s body parts from the video frames. These frames, along with their corresponding masks, are then passed to the ProPainter [30] model to inpaint the occluded human body regions. An example of the data preprocessing pipeline is illustrated in Figure 2. Although the inpainting model is not pretrained on robot-specific data and may not achieve perfect removal, our experiments demonstrate that the resulting demonstrations are sufficient for training high-performance policies.

Using the preprocessed demonstrations, we train Diffusion Policy [44] conditioned on inpainted visual observations, tactile sensing, and proprioception to predict force-informed actions, modeled as $p(x_{d},f_{d}\mid o_{t},q_{t},x_{t})$. Specifically, for the calculated fingertip force vector $\mathbf{f}=(f_{x},f_{y},f_{z})$, we supervise policy training using the normal force component $f_{z}$, which corresponds to the primary axis along which the fingertip can actively exert force, as illustrated in Figure 2.

At inference time, the policy produces action chunks [52] to enable smoother control. Each chunk specifies the desired joint positions and fingertip contact forces, which are executed using the force control strategy described below. Further details of the network architecture and policy training configurations are provided in Appendix B.

In conventional setups, robots are typically operated under position control [53], where the control signal $u$ is computed by a PD controller [54] based on the joint position and velocity errors relative to the target joint positions $x_{d}$:

where $K_{p}$ and $K_{d}$ denote the proportional and derivative gains, respectively.

However, relying solely on position control may be insufficient for certain precise, contact-rich tasks, such as cap twisting or toothpaste squeezing. This is due to policies that only track target joint positions results in merely contacting the object’s surface without applying meaningful forces, as the recorded fingertip positions remain unchanged regardless of the forces applied during kinesthetic teaching. This discrepancy often leads to unstable grasps, slipping, or ineffective manipulation.

To address this limitation, we exploit the following physical property: when a fingertip contacts an object, any nonzero position error continuously generates pressure against the surface due to the object’s resistance, with larger errors producing greater forces, as if the target position lies inside the object. We refer to this virtual displacement as the force-informed target position. To compute it, we utilize the predicted fingertip forces $f_{d}$ from the trained policy, which are directed orthogonal to the contact surface and align with the primary motion axis of the finger joints. Specifically, for each finger, let $x^{tip}$ and $x^{base}$ denote the current positions of the fingertip and base joints, respectively. The force-informed target positions, $x_{d}^{tip}$ and $x_{d}^{base}$, are then computed as:

Here, $K^{tip}$ and $K^{base}$ are two hyperparameters that determine the motion stiffness at the fingertip and base joints. They are tuned to ensure that the execution faithfully tracks the predicted forces and are kept fixed across different tasks. With this force control strategy, KineDex can precisely track the target fingertip forces predicted by the trained policy, thereby achieving stable and force-informed control during execution.

Hardware Setup. For this set of experiments, we implement KineDex using a Franka Emika Panda robotic arm equipped with a Robotera XHand1¹¹1https://www.robotera.com/goods/2.html dexterous hand. Each finger on the XHand1 has two joints, while the thumb and index finger include an additional rotational joint, resulting in a total of 12 degrees of freedom. Each finger is equipped with 120 tactile sensing points.

Baselines. We compare KineDex against three ablated variants: (i) w/o Force Control, which disables force control during inference while keeping the training setup unchanged. (ii) w/o Tactile Input, which removes tactile sensing from the policy inputs during training; however, the policy still predicts target fingertip forces, which are executed using the same force control strategy. (iii) w/o Inpainting, which omits the inpainting preprocessing step for kinesthetic demonstrations.

We evaluate performance by conducting 20 trials per task, with results summarized in Table 1. KineDex achieves over 70% success rates on most tasks, and nearly 100% on common pick-and-place scenarios such as Bottle Picking and Cup Picking. While performance is slightly lower on the final three, more challenging tasks, the average success rates still exceed 50%. This drop is likely due to the increased demands for fine-grained localization and contact reasoning, which may exceed the representational capacity of the current policy inputs. Despite these challenges, the results demonstrate that kinesthetic demonstrations effectively support visuomotor policy learning across a wide range of daily manipulation tasks, owing to their natural alignment with human behavior and the availability of accurate tactile and force feedback.

The ablation results without force control underscore the importance of incorporating both force measurements during training and force control during inference. In this setting, the average success rate across all tasks drops to just 16.7%, with even simple tasks such as Bottle Picking rarely completed successfully. Without force control, the robotic hand often merely contacts the object surface without applying sufficient pressure, leading to frequent failures in contact-rich tasks. To further examine this effect, Figure 3 visualizes the fingertip-object contact forces during inference. KineDex accurately tracks the desired contact forces predicted by the policy, exhibiting similar magnitudes and temporal patterns. In contrast, without force control, the executed force remains flat and fails to follow the predicted targets, indicating a breakdown in physical interaction quality.

When the policy is trained without tactile input, performance exhibits a moderate drop on most relatively simple pick-and-place tasks. This outcome is expected, as visual information alone is often sufficient to estimate required forces in many scenarios. However, for more contact-intensive tasks such as Cap Twisting, Toothpaste Squeezing, and Syringe Pressing, removing tactile input leads to a significant performance decline, with average success rates decreasing by 26.7%. This suggests that in scenarios with severe visual occlusion or tasks heavily reliant on contact feedback, tactile sensing serves as an effective auxiliary modality that substantially improves task success.

The variant without inpainting yields a zero success rate across all tasks and exhibits unreasonable behaviors during execution. These results confirm that raw kinesthetic demonstrations are infeasible for direct policy training due to out-of-distribution visual observations, and that inpainting provides an effective strategy for mitigating this issue.

We further validate the advantages of KineDex over teleoperation for data collection through comparative experiments. We replicate the Open-TeleVision [15] setup to construct a teleoperation system using a Franka Emika Panda arm equipped with an Inspire Hand²²2https://www.inspire-robots.com/product/frwz/, and track the operator’s hand motion using the Meta Quest 3 headset³³3https://www.meta.com/quest/quest-3/. To ensure a fair comparison, we implement KineDex using the same Inspire Hand in this set of experiments. Detailed setup is provided in Appendix C.1.

We evaluate five tasks that are feasible under teleoperation and report the demonstration success rates during data collection. As shown in Table 2, teleoperation achieves an average success rate of 39%, whereas KineDex consistently achieves near-perfect success across all tasks. The absence of real-time tactile feedback significantly impairs teleoperation performance, particularly in tasks such as Cup Picking, where the operator frequently crushes the paper cup due to excessive force. Moreover, because the operator views the scene through a remote camera in the VR interface, the resulting unnatural visual feedback introduces additional challenges, especially for fine-grained manipulation such as Syringe Pressing. These results suggest that teleoperation requires greater operator expertise and repeated trial-and-error to produce high-quality demonstrations, leading to significantly lower data collection efficiency compared to KineDex.

In addition to success rates, we also measure the time required to collect demonstrations with both methods. As shown in Figure 4, the improvement in efficiency is substantial: on the complex task of Syringe Pressing, KineDex completes each demonstration in roughly half the time required by teleoperation; on the simpler task of Bottle Picking, it takes less than one-third of the time. This efficiency gap is primarily due to the additional time required for the operator to adjust to precise hand poses, as well as the inherent latency and limited responsiveness of the teleoperation system.

To provide a more comprehensive evaluation of KineDex, we conduct a user study using the setup described in Appendix C.2. Participants used both KineDex and the teleoperation system, and subsequently provided feedback on their perceived effectiveness and ease of use. The results, summarized in Figure 5, indicate that KineDex outperforms teleoperation across all evaluation criteria. Specifically, all participants agreed that KineDex enables more accurate tactile data collection and is better suited for complex manipulation tasks, while most found it easier to use than teleoperation.