DexCtrl: Towards Sim-to-Real Dexterity with Adaptive Controller Learning

We utilize model-free Proximal Policy Optimization (PPO) reinforcement learning to obtain oracle policies in simulation. Specifically, for each time step $t$, oracle policy $\pi(a_{t},K_{t}\mid s_{t})$ takes in state $s_{t}$, outputs joint action $a_{t}$ and $K_{t}$ simultaneously. The action $a_{t}$ is executed using a controller with parameters $K_{t}$. The desired joint trajectories at $t$ time step are obtained from $q_{t}^{d}=q^{d}_{t-1}+a_{t}$. The detailed task designs for two contact-rich manipulation tasks are described as follows:

State: States of two tasks $s_{t}\in\mathbb{R}^{219}$ contain observation of object and robot over the last three time steps. Robot information $s^{r}_{t}\in\mathbb{R}^{64}$ for each step includes current joint positions $q^{c}_{t}$, desired joint positions $q^{d}_{t}$ and controller parameters $K_{t}$. Object information $s^{obj}_{t}\in\mathbb{R}^{9}$ contains object pose $p^{obj}_{t}\in\mathbb{R}^{6}$ and object property vector $\mu\in\mathbb{R}^{3}$, including scale, mass and friction. In short, the state can be presented as: $s_{t}\triangleq(s^{r}_{t-2:t},s^{obj}_{t-2:t}),s^{r}_{t}\triangleq(q^{c}_{t},q^{d}_{t},K_{t}),\ s^{obj}_{t}\triangleq(p^{obj}_{t},\mu)$

Reward Reward of two tasks $r_{t}$ mainly contains four parts:

The rotation speed reward $r_{rotation}$ encourages the object to rotate faster along a certain axis until it reaches the targeted maximum speed. The contact reward $r_{contact}$ encourages binary contacts between the object and the fingertips. The smoothness reward $r_{smoothness}$ penalizes sudden changes of robot joint positions and torques. The terminal reward $r_{terminate}$ penalizes objects when falling off the fingertips (Rotation) and moving too far from the initial positions (Flipping). Details of reward parameters can be found in the appendix.

We separately train our student policy into two modules, i.e. an action prediction module for trajectory generation and a control parameter prediction module for adapting control parameters, as not only do they encode fundamentally different aspects of the task, but also we want to prevent control parameters prediction from affecting action prediction.

Historical Information for Distillation Though feasible for task completion, oracle policies cannot be directly transferred to the real world because some primitive information such as object information is not directly accessible in real-world settings. To solve this problem, our method utilizes historical states of robot proprioception to distill primitive information used in oracle policy, enabling the estimation of rotated object properties [6]. Concretely, we use the last ten steps of the current and desired joint trajectories, along with corresponding controller parameters as historical information for policy input.

Module Design To better leverage historical information, we use self-attention to model temporal historical input for action prediction. For control parameter prediction, we use cross-attention where current action serves as query and historical input serves as key and value, modeling the relationship between the current action and historical input. Although the historical input formulations of the two modules remain the same, their meaning is completely different. In action prediction, the input mainly indicates the trend of joint trajectory variation, similar to the previous work [7, 8]. In control parameter prediction, it mainly indicates an approximate relation between joint actions and control parameters, which can be analogized to how humans infer current control parameters decisions based on historical trajectories and previous parameters.

Training and Inference Training processes of the two modules are performed in an open-loop manner, meaning all input data is directly retrieved from the collected simulation dataset. However, during both simulation and real-world inference, our method performs closed-loop behaviors, meaning values of current trajectories are obtained from actual robot sensors. We linearly map the control parameters from simulation to their corresponding values on the real-world system, with only an approximate estimate of upper and lower bounds instead of careful calculation and tuning. Furthermore, we find that adding Gaussian noise to the current trajectory values during student policy training is enough for sim-to-real transfer, even though the training dataset is not collected by a teacher policy with input noise randomization. This finding indicates that certain randomization on observation can be reduced to ease the teacher policy training procedure.

Baselines We compare our method with two main baselines:

• •

  Manual Tuning Similar to [5, 7], this baseline trains a new oracle policy with (1) carefully tuned control parameters based on trajectory output comparison between simulation and real world (2) a small range of randomization added on the controller, and then use this new oracle policy for student policy training. Both the new oracle policy and student policy would not output adaptive controller parameters.

• •

  Ours w/o PD This baseline replaces the adaptive controller parameter prediction module in the student policy with fixed controller parameters same as manual tuning, and only trains the action prediction module using the dataset collected by our oracle policy.

Metrics We quantitatively evaluate the performance of all methods with four metrics [6, 7]:

• •

  Rotation Reward/Radians (RotR) This metric represents the rotation speed of the targeted object around the desired axis. In simulation, it is calculated as the average rotation reward over a trajectory. In the real world, it is calculated by the net rotation of object in radians over a trajectory.

• •

  Time To Fail (TTF) This metric represents the average trajectory length before the object falls off the hand or moves too far from its initial positions in rotation and flipping tasks, respectively.

• •

  Object linear Velocity (ObjVel) This metric represents the average magnitude of object linear velocity per action step. It reflects the stability of the targeted object. This metric is only calculated in the simulation because real-world object velocity can not be easily measured in rotation. It is only evaluated in rotation because such behavior is inevitable in flipping.

• •

  Torque Penalty (Torque) This metric computes the torque penalty reward per time step to measure the energy cost and it is only measured in simulation.

We first test DexCtrl in a simulation environment with and without randomly applying force disturbances on objects. During validation, we randomize 1024 different initial robot poses and set the simulation controllers exactly the same as those used during training, meaning no controller gap is involved. This experiment aims to show whether performance can be improved by simultaneously adjusting action and control parameters even under the same controllers. Table 2 and Table 2 present the quantitative results of baselines and DexCtrl in simulation validation. Compared to the manual tuning baseline, DexCtrl improves performance significantly, especially in RotR and TTF. This demonstrates that our method can effectively stabilize and accelerate the task process with or without disturbance. It’s worth noticing that Ours w/o PD also reaches relatively good performance, indicating the robustness of trajectories generated by our method. However, in the flipping tasks with results in Table 2, Ours w/o PD does not outperform the baseline. We believe it is because flipping involves rich contact between both the floor and the dexterous hands, making it more sensitive to controller parameters’ variance. In conclusion, our method outperforms the baseline in both task performance and training speed, even in the absence of the controller gap.

We directly deploy DexCtrl in real-world scenarios. For the in-hand rotation task, we use twelve different real-world unseen objects with varying masses and frictions, and validate rotation performance based on the average metrics over ten randomly sampled initial robot poses per object. Table 3 and Figure 4 present quantitative results and visualizations of DexCtrl along with baselines across different objects, respectively. Compared to the simulation results, the performance gap among different methods is more pronounced in the real world, where DexCtrl significantly outperforms the baseline under zero-shot sim-to-real transfer. Also, Ours w/o PD achieves relatively strong performance, demonstrating that trajectories generated by DexCtrl can be more robust when transferred to real-world scenarios. It’s worth noticing that the performance gap between DexCtrl and Ours w/o PD is much larger in the real world than in simulation. This finding highlights the necessity of adaptively adjusting control parameters at every step in real-world robots, which further emphasizes the importance of adaptive controller prediction in tackling sim-to-real issues. We also conduct real-world experiments on the flipping task with visualizations shown in Figure 3, demonstrating the real-world task generalizability of our method.

To further evaluate our policy, we perform additional tests on objects with different masses and friction coefficients. To ensure better control over confounding factors, we use a hollow cube for mass experiments and vary its mass by inserting different internal objects, and use cubes of different textures with the same mass for friction experiments. As shown in Table 4 and Figure 5, our method significantly outperforms both baselines, especially on heavy objects, indicating that it can better adapt to objects with different physical parameters. Also, the pattern of our method broadly aligns with our force-based predictions, namely, the lightest and smoothest objects exhibit the highest speed and lowest stability, respectively.

In this section, we investigate the fundamental question: how do the learned controller parameters affect dexterous manipulation performance? To simplify the analysis, we focus solely on the pattern of learned stiffness, as it contributes more than damping to the task performance. Theoretically, changes in stiffness directly influence the resulting joint torques, thereby modulating the contact forces between the robot and the manipulated object. Given that objects with different physical properties (e.g., mass and friction) require distinct contact force profiles, we hypothesize that variations in stiffness are closely related to object-specific configurations, aiming to provide better adaptation to varying force requirements.

To validate this hypothesis, we first analyze data collected in the simulation. Figure 6 illustrates the relationship between the average stiffness observed across trajectories and object mass as well as surface friction. The results reveal that stiffness increases monotonically with mass (left of Figure 6), consistent with the intuitive physical principle that heavier objects require greater force for manipulation. However, the relationship between stiffness and friction is more nuanced: in some cases, stiffness increases with friction (middle of Figure 6), while in others it decreases (right of Figure 6). We attribute this inconsistency to task-dependent dynamics. For instance, in rotation tasks, friction may primarily resist angular motion at certain joints, while at others it may act more like a supporting or pushing force. This suggests that the role of friction—and consequently the required stiffness—depends on both the object’s properties and the specific joint-task interaction.

To further investigate this relationship in real-world settings, we isolate the controller parameters prediction module and provide ground-truth actions to only predict controller parameters. This allows us to observe the influence of object properties without the confounding effects of policy action noise. Figure 7 presents the trends across objects with varying mass and friction. Two consistent patterns emerge: (1) for heavier objects, stiffness tends to increase at specific time steps or remain at its maximum value for longer durations; (2) for smoother objects, stiffness exhibits similar patterns at some joints but opposite trends at others, again indicating task- and joint-specific behavior.

In summary, our results show that the learned stiffness adapts systematically to object mass and friction, validating our hypothesis that controller parameters encode variations in required contact forces and thereby enhance manipulation performance through better adjustment to force requirements.