Generating Task-specific Robotic Grasps

Our three-level object representation is based on a single exemplar for each object class. At the top level, any object is an instance of a particular class (e.g., cup or hammer). The lowest level is a point cloud representation of an object’s surfaces, encoding the geometric data (e.g., shape, size) necessary for planning grasps. The intermediate level is a small set of keypoints that concisely define the object’s pose while allowing for in-class variations in shape and size.

The point cloud representation of an object is based on the probabilistic signed distance function (pSDF) [25], which allows multiple views of an object to be combined and models measurement uncertainty. A voxel grid is used to capture the signed distance to the surface from the center of each voxel. Distance data is only stored for voxels close to the measured surface using a truncation threshold. Instead of modeling distance uncertainty as a weight [26], surface distance is modeled as a random variable with a normal distribution $\mathcal{N}(\overline{d}_{sdf},\,\sigma_{sdf}^{2})$. The effect of any new measurement ($d_{sens},\sigma_{sens}^{2}$) at step $k$ is merged using a Gaussian update:

where the measurement variance ($\sigma_{sens}^{2}$) is computed experimentally (see Equation 14). We implement this encoding by revising the Open3D library [27]. A uniformly-sampled point cloud, with an uncertainty measure at each point, is then extracted from the pSDF representation using the marching cubes algorithm [28].

Our intermediate representation is a lightweight set of semantic keypoints that builds on existing work [24, 4] to model large in-class variations—Figure 2(b) illustrates this for screwdrivers—and provide an object pose estimate in terms of the spatial relationships between keypoints. In our implementation, 2D keypoints are detected in each colour image using a stacked hourglass network originally developed for human pose estimation [29]. We triangulate the 3D positions of keypoints from multiple views, with links between the keypoints forming a skeleton. This skeleton is used to guide grasp generation and (as we show in Section 4.2) to transfer knowledge for grasping across similar objects when used to perform similar tasks.

Our objective is to identify good grasps while considering stability, measurement uncertainty, and task-specific constraints. There is existing work on considering different criteria to design and evaluate grasps with different grippers, e.g., estimating the marginal success probability $P(S)$ of a proposed grasp $g$, given object state $x$, as the sum of a weighted set of probabilistic criteria $P_{i}$ [30]:

where $w_{i}$ are the weights. We modify the original criteria to incorporate task-specific scoring functions and focus on factors relevant to any dexterous gripper. Additional functions and factors can be added to modify grasping behaviour for specific gripper designs or tasks.

One factor of interest is a good contact angle $\alpha$ between gripper finger contact vector and the surface normal of the contact region on the object. We estimate the probability of good contacts for grasps with any number $k$ of finger tips:

where $\theta$ is the friction cone’s maximum contact angle.

The next factor is based on the insight that the extent to which an object’s surface is recovered from observed data contributes to the grasp success probability. Poor surface recovery makes it difficult to estimate the surface location, and a surface with a high degree of curvature or variations may not provide a good contact region. The probability of a set of surface points being good contact points is given by:

where $u$ is a measure of the surface variation at each point based on the eigenvalue decomposition of points in the local neighbourhood; $u=0$ if all points lie on a plane and $u=1/3$ if points are isotropically distributed [31]. Also, $c$ is an uncertainty estimate for the point from the pSDF representation with $c_{max}$ as the maximum value; the variance is determined by experimentally set parameters.

Finally, each task and object-specific constraint is encoded as a function that assigns a score $\in[0,1]$ to each recovered surface point describing its likelihood of being a finger tip location for grasps that meet this constraint. The overall probability for this factor is then the product of the values of probabilistic functions for individual constraints:

In this paper, we illustrate this factor by considering one constraint per combination of task and object. Next, we describe these probabilistic functions.

While defining the functions that encode task and object-specific constraints about where to grasp an object, the objective is to model and preserve the relationship between keypoints and suitable grasp points for specific tasks across changes in factors such as scale and orientation.

Specifically, we extend the keypoint representation to include a class-specific Euclidean coordinate system at each keypoint—see Figure 2(b). Consider a pair of keypoints on an object for which an exemplar grasp is available in the form of recovered grasp points on the object’s surface for any particular task. The keypoints are linked by a line segment of length $S$. The origin of the Euclidean coordinate system would be at one keypoint (the reference) with the x-axis aligned with the link. The orientation of the y-axis and z-axis axes are defined by the plane formed by these two keypoints and one other keypoint, and the normal to the plane; if the object model has less than three keypoints, the coordinate system is based on eigenvectors of the object’s point cloud.

Given the axes at a keypoint, unit directional vectors $\langle x,y,z\rangle=1$ are computed to each recovered surface point of the exemplar grasp. These vectors are scaled onto a sphere of radius $S$ and converted into spherical coordinates $\theta,\phi$ with:

where $\theta$ $\in[-2\pi,2\pi]$ and $\phi$ $\in[-\pi,\pi]$. The scaled vectors are used to construct a Gaussian Mixture Model (GMM) for each keypoint using an existing software implementation of Dirichlet Process inference and Expectation Maximisation [32]. Our representation based on spherical coordinates provides some robustness to scaling and orientation changes in new object instances of the corresponding class. At run time, point cloud points on the surface of an object are assigned a probabilistic score based on the GMM models of the “relevant” keypoints, i.e., those that are on the nearest link (based on Euclidean distance). The object and task-specific probability value $T_{i}$ for each surface point is the product of the probabilities from the relevant keypoints.

We developed a sampling-based grasp generation algorithm. For any target object, RGBD images collected from four predefined poses are combined using the pSDF method described in Section 3.1 to obtain the object’s point cloud, with the measurement error estimated as described in Section 4.1. A set of $45$ starting positions are sampled on the object’s recovered surface (i.e., point cloud) with sample probability $T_{i}$ (computed as described above). Note that the baseline algorithm (for experimental evaluation) does not consider task knowledge, i.e., all points are equally likely to be sampled.

For each sampled point, the robot virtually simulates the placement of a fingertip aligned to oppose the surface normal at the point. The gripper is positioned so that the body of the gripper is located above the object at rotations of 0^∘ and +/-20^∘ from the plane formed by the surface normal (at the sampled point) and the upward vector of the robot’s base coordinate system. We use the Eigengrasp approach [10] to close the gripper around the object until the other fingers contact the object’s surface. The contact points of the grasp candidates which successfully close to form additional contacts with the object, and which do not collide with the table plane, are assigned scores using the weighted criteria described in Section 3.2. Of the 135 samples, the best scoring grasp candidate in the virtual experiments is further optimised locally in three iterations with initial offsets in grasp rotation and position of 15^∘ and 5mm, which are halved in each iteration. Only the best (optimised) grasp is executed on the robot. Note that this method is applicable to different gripper designs with different number of fingers.

Physical experiments were conducted using a Franka Panda robot manipulator with low-cost Intel Realsense D415 cameras mounted on the end effector and as shown in Figure 1. Objects were placed on the gray tray to ensure it was within the robot’s workspace and field of view. We segmented the object from the background in each image using a pretrained Detectron2 segmentation network [33]. The masked RGBD images were passed through the pSDF algorithm to obtain a point cloud.

Figures 3 and 4 illustrate the use of some task models from two object classes (screwdriver, cup), including the scoring of surface points of five previously unseen objects from these classes. We show just the point cloud data in these figures for ease of explanation; during run time, our algorithm processed real-world scenes containing the objects of interest.

In Figure 3, we can see that the learned models of task and object-specific constraints scale well to new object instances despite variations in the shape of the handle or length of the screwdriver shaft. The nonlinear shapes of the cups is modelled and considered when evaluating the suitability of new surface points as grasp points in Figure 4, with different sets of points being preferred for the the handover task and the pour task. These qualitative results demonstrate our method’s ability to use models learned from a small set of exemplars to evaluate grasps for different tasks and guide grasping towards locations favoured by the domain expert.

To evaluate H1-H3, we completed a total of 530 grasp trials on the physical robot platform. Table 1 summarizes results for all five classes. The results for the object class Hammer are shown separately in Figure 5 that focuses on the stability criterion to highlight some interesting results that are discussed further below.

In Table 1, bold-faced numbers along each row indicate the best scores for the corresponding object class for each of the three tasks. For example, for the class cup, the ’Baseline’ and ’Handover’ model were equally good for providing good stability, while the ’Handover’ model and ’Tool use’ model (i.e., the task-specific models) provide the best performance for the corresponding tasks. Our experiments showed that for some object classes the baseline algorithm is more likely to produce grasps which suit one task, e.g., handover for cups and tool use for screwdrivers. This result is expected as these tasks require grasps which place the fingers on larger, lower curvature areas of the model that better fit the stability criteria optimised by the baseline approach. Results also indicated that including task and object-specific information in the learned models for different classes steered grasps towards regions that better suit the task under consideration and produced more successful grasps.

To evaluate H2, we focused on the brush and dustpan object classes, which have a “handle” configuration similar to that of the screwdriver; we modelled this configuration with a keypoint at each end. In our trials for these new object classes, we applied the same “tool use” model which had been trained for the screwdriver. Our results show that this produced grasps with both high stability and task suitability illustrating that our method can be used to translate learned task knowledge to other similar object classes when they are used to perform similar functions.

The results also supported H3. For example, experiments in the screwdriver class showed adaption to scaling changes by stretching the learned models based on keypoint positions to handle length (size) differences. Also, the trials for the cup class illustrated the ability of our approach to handle intra-class variations in appearance and shape, with each cup having different height, diameter, thickness, and handle design. Figures 3 and 4 show some variations in these two classes in the point cloud representation; additional qualitative results are shown in Figure 6.

For some objects there is a small drop in stability with our approach when our task-specific models result in the robot targeting regions that the baseline algorithm is less likely to target, such as when the robot attempts to grasp the cup handle for pouring. This is expected as the handle regions typically have reduced low curvature surface area for the gripper finger tips to contact. The drop in stability is balanced by the other constraints in grasp scoring resulting in grasps with an acceptable stability which also meet the task requirements. As an additional test, we added a 6mm thick strip of foam to the handle of the cup with the thinnest handle and repeated an additional ten trials with the pour task model. With a larger surface area for grasping, the rate of successful grasps increased from 70% to 90% with our approach, and the proportion of these grasps that met the criteria for the pour task increased from 71% to 88%. Overall, these results support hypotheses H1-H3.

The results in Figure 5 for the hammer class highlight some issues of interest. Many grasp algorithms in literature aim for the centre of objects to increase the likelihood of a stable grasp. In the case of a hammer this heuristic is flawed as the functional design of a hammer requires concentration of mass at the head end. With the baseline model and the parallel gripper, it was difficult to find a stable grasp to lift a hammer without the hammer rotating in the grasp; the robot was not able to successfully grasp, lift, and hold the hammer for the required time in many trials. The use of our learned models showed a significant improvement in stability over the baseline method because our approach learned to guide grasping away from the handle and towards the head end of the hammer shaft for stability (and handover). The functional distribution of weight and our choice of gripper make it difficult to run trials successfully for a tool use task even with our learned model; the robot often selects the correct grasp points on the handle but is unable to stably hold the object for the required time. This can be fixed by using a different gripper (e.g., more fingers that wrap around the the handle), which we will explore further in future work.