Cloth Manipulation Using Random-Forest-Based Imitation Learning

Our main goal is to optimize the learnable parameters $\beta$ to optimize the performance of the controller, $\pi$. This controller optimization problem can take different forms depending on the available information about $\mathbf{c}^{*}$. If $\mathcal{O}(\mathbf{c}^{*})$ is known, then we can define a reward function: $R(\mathbf{c})=-\mathbf{dist}(\mathcal{O}(\mathbf{c}),\mathcal{O}(\mathbf{c}^{*}))$, where $\mathbf{dist}$ can be any distance measure between RGB-D images. In this setting, we want to solve the following reinforcement learning problem:

where $\tau=(\mathbf{c}_{1},\mathbf{c}_{2},\cdots,\mathbf{c}_{\infty})$ is a trajectory sampled according to $\pi$, $\gamma$ is the discount factor, and the subscript figures denote the timesteps. Another widely used setting assumes that $\mathcal{O}(c^{*})$ is unknown, but that an expert is available to provide an optimal control action $\pi^{*}(\mathcal{O}(\mathbf{c}))$. The expert is a ground truth controller following the definition of [23]. In this case, we want to solve the following imitation learning problem:

This expert can be easily acquired in a typical human-robot collaboration task. Our method is based on the imitation learning formulation.

Before constructing the random-forest from $\mathcal{D}$, we apply a feature transform to $\mathcal{D}$. Our raw observation of the cloth, $\mathcal{O}(\mathbf{c})$, is an RGB-D image. it has been noted, (e.g., by [36]) that applying a simple feature transform can improve the accuracy of a classifier such as random-forest. In addition, our input is a $320\times 240$ RGB-D image of the cloth mesh, which corresponds to $76800$ entries each having three colors and one depth channel, which is high-dimensional. Therefore, a feature transform effectively reduces the dimensions of the input observation and makes the classifier more robust when the size of the dataset is small.

In our approach, we use HOW-features [7] as the low-dimensional representation. HOW-features is a variant of HOG-features. HOW-features first applies Gabor filters to each patch of the image and then concatenates these patches, resulting in a $768$-dimensional feature space. Since each image patch is spatially localized, HOW-features requires each image to be aligned as a pre-processing step. Because our input is an RGB-D image, we can perform a foreground extraction using the depth-channel and then align the image to the center of the screen using the same procedure as in [36]. We summarize this algorithm in Algorithm 1 and denote this feature transform as a function $\mathcal{F}$. The dataset after the feature transform is defined as $\bar{\mathcal{D}}=\{\langle\mathcal{F}\circ\mathcal{O}(\mathbf{c}),\mathbf{x}^{*}\rangle\}$.

Our key contribution is to use a random-forest as the underlying learnable controller in an imitation learning framework. A random-forest is an ensemble of $K$ decision trees, where the $k$-th tree classifies $\mathcal{F}\circ\mathcal{O}(\mathbf{c})$ by bringing it to a leaf-node $l_{k}(\mathcal{F}\circ\mathcal{O}(\mathbf{c}))$, where $1\leq l_{k}(\mathcal{F}\circ\mathcal{O}(\mathbf{c}))\leq L_{k}$ and $L_{k}$ is the number of leaf-nodes in the $k$-th decision tree. The random-forest makes its decision by classifying $\mathcal{F}\circ\mathcal{O}(\mathbf{c})$ using every decision tree and then computing the average over all the decisions of the trees in the forest. To use an already constructed random-forest as a controller, we define an optimal control action $x_{l,k}^{*}$ so that the final action is determined by averaging:

To construct the random-forest, we use a strategy similar to that in [37]. We construct $K$ binary decision trees in a top-down manner, each using a random subset of $\mathcal{D}$. Specifically, for each node of a tree, a set of random partitions is computed and the one with the maximal Shannon information gain [38] is adopted. Each tree is grown until a maximum depth is reached or the best Shannon information gain is lower than a threshold. The optimal control action of a leaf-node is defined as the average of the control actions of the data sample belonging to that leaf-node.

We use an imitation learning algorithm [27] that includes two steps into an outer loop. During each outer iteration, we query an expert, which in our case is a ground-truth hard-coded control algorithm. Specifically, we generate a set of cloth simulation trajectories using a cloth simulator (Equation 2). During each timestep of these trajectories, we query the expert to get an optimal control action $\pi^{*}(\mathcal{O}(\mathbf{c}))$. This optimal control action is combined with the action proposed by our random-forest $\pi(\mathcal{O}(\mathbf{c}))$. The combined action is fed to the simulator to get the next observation. As a result, more data is added into $\mathcal{D}$ and a new random-forest, $\beta$, is constructed from a new $\mathcal{D}$. This algorithm is outlined in Algorithm 2.

In typical DOM applications, data are collected using numerical simulations. Unfortunately, the high dimensionality of $\mathbf{c}$ induces a high computational cost for simulations (i.e. evaluating $P$ in Equation 2) and generating a large dataset can be quite difficult. Therefore, we design our method so that it can be used with a small number of data samples. Our method’s performance relies on the random-forest’s stopping criterion (i.e. the threshold of gain in Shannon entropy). We choose to use a large Shannon entropy threshold so that the random-forest construction stops early, leaving us with a relatively small number of leaf-nodes. We expect that, with a large enough number of imitation learning iterations, the number of nodes in each decision tree of the random-forest will converge. Indeed, such convergence can be guaranteed by the following Lemma.

Lemma: When the number of imitation learning iterations $N\to\infty$, the distribution incurred by the random-forest-based controller will converge to a stationary distribution and the expected classification error of the random-forest will converge to zero.

Proof: By assuming that Algorithm 2 generates a controller $\pi^{n}$ at the $n$-th iteration, Lemma 4.1 of [27] showed that $\pi^{n}$ incurs a distribution that converges when $n\to\infty$. Obviously, the number of data samples used to train the random-forest also increases to $\infty$ with $n\to\infty$. The expected error of the random-forest’s classification on a stationary distribution converges to zero according to Theorem 5 of [39]. In Section V, we show that, empirically, the number of leaf-nodes in the random-forest also converges to a fixed value.

We evaluate our method on a simulated environment. For the simulated environment, the robot’s kinematics are simulated using Gazebo [41] and the cloth dynamics are simulated using ArcSim [35], a highly accurate cloth simulator. We use OpenGL to capture RGB-D in this simulated environment. Our goal is to manipulate a 35cm$\times$30cm rectangular piece of cloth with four corners initially located at: $v^{0}=(0,0,0),v^{1}=(0.3,0,0),v^{2}=(0,0.35,0),v^{3}=(0.3,0.35,0)$(m). Our manipulator holds the first two corners, $v^{0},v^{1}$, of the cloth and the environmental uncertainty is modeled by having a human hold the last two corners, $v^{2},v^{3}$, of the cloth so that we have $\mathbf{x}\triangleq(v^{0},v^{1})^{T}$ and each control action is $6$-dimensional. The human could move $v^{2},v^{3}$ to an arbitrary location under the following constraints:

where the first constraint avoids tearing the cloth apart and the second constraint ensures that the speed of the human hand is slow.

To evaluate the robustness of our method, we design the 3 manipulation tasks listed below:

• •

  Cloth should remain straight in the direction orthogonal to human hands. This is illustrated in Figure 3 (a). Given $v^{2},v^{3}$, the robot’s end-effector should move to:

  $\displaystyle v^{0}=v^{2}+0.35\frac{z\times(v^{3}-v^{2})}{\|z\times(v^{3}-v^{2})\|}\;v^{1}=v^{3}+0.35\frac{z\times(v^{3}-v^{2})}{\|z\times(v^{3}-v^{2})\|}.$

• •

  Cloth should remain bent in the direction orthogonal to human hands. This is illustrated in Figure 3 (b). Given $v^{2},v^{3}$, the robot’s end-effector should move to:

  $\displaystyle v^{0}=v^{2}+0.175\frac{z\times(v^{3}-v^{2})}{\|z\times(v^{3}-v^{2})\|}\;v^{1}=v^{3}+0.175\frac{z\times(v^{3}-v^{2})}{\|z\times(v^{3}-v^{2})\|}.$

• •

  Cloth should remain twisted along the direction orthogonal to human hands. This is illustrated in Figure 3 (c). Given $v^{2},v^{3}$, the robot’s end-effector should move to:

  	$\displaystyle v^{0}=\frac{v^{2}+v^{3}}{2}+0.31\frac{z\times(v^{3}-v^{2})}{\|z\times(v^{3}-v^{2})\|}+0.15\frac{(v^{3}-v^{2})\times(z\times(v^{3}-v^{2}))}{\|(v^{3}-v^{2})\times(z\times(v^{3}-v^{2}))\|}$
  	$\displaystyle v^{1}=\frac{v^{2}+v^{3}}{2}+0.31\frac{z\times(v^{3}-v^{2})}{\|z\times(v^{3}-v^{2})\|}-0.15\frac{(v^{3}-v^{2})\times(z\times(v^{3}-v^{2}))}{\|(v^{3}-v^{2})\times(z\times(v^{3}-v^{2}))\|}.$

The above formula for determining $v^{0},v^{1}$ is used to simulate an expert. Note that these equations for the expert requires the knowledge of the four corner positions of the piece of cloth, and such information may not be available in a real robot system that only observes the cloth using a single RGB(D) image. Therefore, we train our random-forest in a simulated environment. These three equations assume that the expert knows the location of the human hands, but that robot does not have this information and it must infer this latent information from a single-view RGB-D image of the current cloth configuration. We also test the performance on complex benchmarks that combine flattening, folding, and twisting, or have considerable occlusion from a single camera.

Although we have only evaluated our method on a simulated environment, we can also deploy our controller on real robot hardware. For the real robotic environment, we use a RealSense depth camera to capture 640$\times$480 RGB-D images and a 12-DOF ABB YuMi dual-armed manipulator to perform the actions, as illustrated in Figure 3.

To deploy our controller, we first use camera calibration techniques to get both the extrinsic and intrinsic matrix of the RealSense Camera. Second, we compute the camera position, camera orientation, and the clipping range of the simulator from the extracted parameters. Third, we generate a synthetic depth map using these parameters and train the three tasks using the random-forest-based controller parametrization and the imitation learning algorithm. Finally, we randomly perturb the human hand positions when collecting training data to make our random-forest robust to observation noises. A similar technique is used in [42]. We also add visual noises to the training samples and test the algorithm by posing objects between camera and object. After that, we integrate the resulting controller with the ABB YuMi dual-armed robot and the RealSense camera via the ROS platform. As shown in Figure 1, with these identified parameters we can successfully perform the same tasks that were performed in the synthetic benchmarks on the real robot platform.

Unlike single-task controller, a multi-task random-forest-based controller stores multiple actions in a leaf-node. Each observed image is classified by each decision tree in a manner that is similar to that of a single-task controller. The leaf node chooses an action according to the id of the task. In this benchmark, we train a 3-task controller for the 3 synthetic tasks in Section V-B. And we transfer the controller to the real robot as benchmark (5) mentioned in Figure 1. We combines straightening, bending and twisting to show that our approach can perform complex tasks, as shown in the video. Moreover, we also show tasks which involve occlusion from a single camera viewpoint by adding noise to inputs.

We compare the performances of a single-task controller and a multi-task controller, both of which are based on random-forests. Again, during each evaluation in the simulated environment, the human hands move to $10$ random target positions $v^{2*},v^{3*}$. As shown in Figure 6 (red), we profile the residual (Equation 8). Our controller performs consistently well with a relative action error of $0.4954$%. We then train a joint 3-task controller. This is performed by defining a single random-forest and defining $3$ optimal actions on each leaf-node. The performance of the 3-task controller is compared with that of the single-task controller in Figure 6. The multi-task controller performs slightly worse in each task, but the difference is quite small.

As illustrated in Algorithm 2, the complexity of our overall approach mainly depends on three parts: dataset sampling, feature extraction, and random-forest construction. When constructing a single decision tree based on the sampled dataset $\bar{D}$, the complexity has an upper bound of $O(|\bar{D}|^{2})$. For the construction of a random-forest with $K$ decision trees, the complexity is $O(K|\bar{D}|^{2})$.

To evaluate the performance of each component in our method, we run several variants of Algorithm 2. All the meta-parameters used for training are illustrated in Table II. In our first set of experiments, we train a single-task random-forest-based controller for each task and profile the mean action error:

with respect to the number of imitation learning iterations (Line 4 of Algorithm 2). As illustrated in Figure 7 (red), the action error reduces quickly within the first few iterations and later converges. We also plot the number of leaf-nodes in our random-forest in Figure 7 (green). As more iterations are performed, the number of leaf-nodes in our random-forest also converges.

A key feature of our method is that it allows the robot to react to random human movements while the effect of these movements is indirectly reflected via a piece of cloth. This setting is similar to [43]. However, [43] assumes the 3D geometric mesh of cloth $c$ is known without any sensing error, which is not practical.

Our method falls into a broader category of visual-servoing methods, but most previous work in this area (such as [44]) has focused on navigation tasks and there is relatively little work on deformable body manipulation. [45] based their servoing engine on histogram features, which is similar to our use of HOW-features. However, they use direct optimization to minimize the cost function ($\mathbf{dist}(\mathcal{O}(\mathbf{c}),\mathcal{O}(\mathbf{c}^{*}))$), which is not possible in our case because our cost function is non-smooth in general.

Finally, our method is closely related to methods in [16, 17], which also use random-forest and store actions on the forest. However, our method is different from prior methods in two ways. First, our controller is continuous in its parameters, which means it can be trained using an imitation learning algorithm. Moreover, we use both feature extraction and controller parametrization in the imitation learning algorithm [27] so that both the feature extractor and the controller benefit from evolving training data.

To show the benefits of random-forest, we compare three different models of controllers: random-forest, linear regression, and neural network [27]. During each evaluation in the simulated environment, the human hands move to $10$ random target positions $v^{2*},v^{3*}$. In Table III, we plot of the residual (Equation 8) of the tree methods against the number of imitation learning iterations. On the convergence of Algorithm 2, the random-forest-based controller outperforms the two other opponents, exhibiting a lower residual.

To implement the neural-network-based controller, we use Tensorflow, which is a neural network toolkit. The structure of the neural network is fully connected and consists of a hidden layer of 128 neurons. To implement the linear-regression-based controller, we use the apply the implementation from scikit-learn [46], which is a standard machine learning toolkit. We use the standard parameters from the linear regression module.

There are many standard techniques for computing low-dimensional controlling parameters from high-dimensional perceptual data such as RGB images and depth maps. These include standard regression models and neural-network-based models. We evaluate the performance of our algorithm along with the others. The test involves measuring the residual of the manipulator as it moves towards the goal configuration based on the computed control parameters, as given by Equation 8.

We obtain best results in our benchmarks using a random-forest-based controller. Using the random-forest-based controller and the imitation framework requires fewer parameters to configure a task. Further, the computed control parameters are limited to the labels of the random-forest, which makes the controller robust to the unseen data. In practice, the random-forest-based imitation learning requires fewer computation resources which can enable the controller to be used in real-time applications. The performance is governed by the total number of iterations of the imitation learning. As the number of iterations of imitation learning grows, the residual Equation 8 reduces. After reaching a certain iteration, the imitation learning contributes less to the performance enhancement. In other words, when the imitation learning framework converges, the overall performance of the controller is guaranteed.