SIS: Seam-Informed Strategy for T-shirt Unfolding

There are three main types of strategies for selecting grasping points in garment unfolding or other garment handling tasks: heuristic-based strategy, matching-based strategy, and value-map-based strategy.

One of the key issues for learning-based methods is the lack of data sets when dealing with garments due to the difficulty of annotation. To address this problem, most researchers [8, 9, 10, 11, 12, 18] train their networks using simulators. SpeedFolding [17] uses real data annotated by humans together with self-supervised learning. In [4] and [6], the use of color-marked garments is proposed to automatically generate annotated depth maps. The use of transparent fluorescent paints and ultraviolet (UV) light is proposed by [19] to generate RGB images annotated by the paints observed under UV light.

In addition to the commonly used Pick&Place, the Flinging proposed by FlingBot[8] is widely used for unfolding actions. FabricFolding[12] introduces a Pick&Drag action to improve the unfolding of sleeves. Meanwhile, DextAIRity[20] proposes the use of airflow provided by a blower to indirectly unfold the garment, thereby reducing the workspace of robots. UniFolding[11] uses a Drag&Mop action to reposition the garment when the grasping points are out of the robot’s reach.

Given an observation $o_{\tau-1}\in\mathbb{R}^{W\times H\times 3}$, captured by an RGB camera with resolution of $W\times H$ at episode step $\tau\in\{1,2,...\}$, our goal is to develop a policy $\Pi$ mapping $o_{\tau-1}$ to an action $a_{\tau}$, i.e. $a_{\tau}=\Pi(o_{\tau-1})$, to make the garment converge to a flattened configuration. The action $a_{\tau}$ can be described by the pair of grasping points for the left and right hands, ${}^{\mathrm{img}}\textbf{{p}}^{\mathrm{L}}_{\tau}\in\mathbb{R}^{2}$ and ${}^{\mathrm{img}}\textbf{{p}}^{\mathrm{R}}_{\tau}\in\mathbb{R}^{2}$, w.r.t the image frame $\Sigma_{\mathrm{img}}$.

Existing strategies try to map the $o_{\tau-1}$ to a canonical space or a value map to select grasping points pixel-wise. This paper proposes a new Seam-Informed Strategy (SIS) to select grasping points based on seam information. To implement the mapping policy $\Pi$ based on SIS, we divide the policy $\Pi$ into two methods, $\omega$ and $\pi$, as shown in Fig. 2.

The method $\omega$ called the Seam Feature Extraction Method (SFEM) is proposed to extract seam line segments and their crossings from $o_{\tau-1}$. We use the extracted set of seam line segments $\textbf{F}^{\mathrm{S}}_{\tau}$ and the set of seam crossing segments $\textbf{F}^{\mathrm{C}}_{\tau}$ as candidates for grasping points. The method $\omega$ is formulated as follows:

The method $\pi$ called the Decision Matrix Iteration Method (DMIM) is proposed to select a pair of grasping points from the candidates obtained above for the unfolding action of the dual-arm robot. The decision matrix takes into account both human prior knowledge and the results of real robot executions. The method $\pi$ is formulated as follows:

From (1) and (2), we have:

The proposed SFEM consists of a seam line-segment extractor (SLE) and two seam crossing-segment detectors (SCD1 and SCD2) as shown in Fig. 2 which are designed based on the YOLOv3 [21]. Extracting seams as segments allows them to be used as grasping point candidates.

The idea of DMIM is proposed based on the observations of human unfolding actions using a flinging motion:

• •

  The Combination of Seam Segment Types (CSST) of two selected grasping points affects the unfolding performance. For example, grasping two points both at the shoulders or bottom hems results in a higher unfolding quality than the other CSSTs.

• •

  The current step of the unfolding action affects the unfolding performance of subsequent actions, since an appropriate action simplifies the complexity of the subsequent steps. To obtain a simpler configuration for subsequent actions, we intuitively select two grasping points that are empirically far apart.

Based on these observations, we extract the human skill of unfolding by scoring the performance of each CSST through human demonstrations. The DMIM proposed in this paper represents the grasping strategy using a decision matrix. The performance score of each CSST is implemented by initializing the decision matrix with human demonstrations and updating it with robot executions.

Below are the major motion primitives associated with the T-shirt unfolding experiments.

In this section, we perform ablation studies for our proposed strategy. In the following experiments, 20 trials are performed using two trained clothes in Category T in Fig. 7, with each trial consisting of five episode steps. The results are shown in Table I and Fig. 5. We use normalized coverage [8, 17, 20] and IoU between the current configuration and the manually unfolded configuration [9] as evaluation metrics. Fig. 5 shows the distributions of normalized coverage with 95% confidence interval shading.

In Table I, the Full group is the complete scheme proposed in this paper based on seam information and DMIM using $\textbf{{U}}^{\mathrm{D}}$.

The Ab-SI group is carried out without the seam information extracted by the SLE and uses only the crossings detected by the SCD2 as candidate grasping points, while $\textbf{{U}}^{\mathrm{D}}$ is used. The difference between the Ab-SI and Full groups in Fig. 5 (a)(b) shows that the seam information significantly improves the unfolding performance.

The Ab-MI group is performed to show the effectiveness of the proposed iteration method. In this experiment, $\textbf{{U}}^{\mathrm{D}}_{\mathrm{init}}$ is used without updating. Fig. 5 (c)(d) shows the human-to-robot gap in the performance of the unfolding action. The results of the Full and Ab-MI groups show that our proposed DMIM bridges the human-to-robot gap.

We conduct a right-tailed Welch’s t-test to analyze the differences in the means of the ncov and IoU metrics before and after removing one component from the full scheme. Each group consists of 20 samples at each episode step. The null hypothesis $H_{0}$ posits that the Full group does not outperform the ablated configuration. The alternative hypothesis $H_{1}$ posits that the Full group significantly outperforms the ablated configuration. We reject the null hypothesis $H_{0}$ in favor of the alternative hypothesis $H_{1}$ when the p-value $p$ is less than the significance level $\alpha=0.05$, concluding that the performance of the Full group is significantly superior to that of the Ab-SI or Ab-MI group.

As shown in Table II, removing the Seam Information (SI) component results in statistically significant decreases in ncov during the early steps ($p<0.05$). Similarly, removing the Matrix Iteration (MI) component leads to statistically significant decreases in ncov in later steps, with some decreases being highly significant ($p<0.01$). The IoU results exhibit similar trends. Notably, both groups show significant performance drops during the second step, highlighting the advantages of our proposed method in addressing challenging scenarios, particularly when the garment is in a crumpled state.

Note that the following hardware-related failures are excluded from the statistical data since they are not relevant to the performance of our proposed strategy.

• •

  Grasping failures: We consider a grasp to have failed if either one or both of the grippers failed to grasp the selected grasping points on the T-shirt. We also consider the grasp to have failed if the T-shirt falls off the gripper(s) during the robot motion. Note that similar to [8], we have not filtered out the data when the T-shirt is grasped in its crumpled state, which makes the flinging motion ineffective.

• •

  Motion failures: These occur when the grasping points exceed the working space of the robot, or when an emergency stop is triggered due to robot singularity.

• •

  Releasing failures: We consider a garment release to have failed if the garment remains in the gripper after the gripper fingers open.

To evaluate the generalization ability of our proposed scheme, we conducted experiments with four unseen short-sleeved T-shirts in Category A in Fig. 7. In Fig. 6 (a), we plotted the average result of normalized coverage in 20 unfolding trials, each with five episode steps. Five continuous unfolding trials are performed for each of the four T-shirts. On average, the normalized coverage reaches over 0.865, 0.909, and 0.894 within three, four, and five steps, respectively.

We compare the proposed method in this paper with those presented in [8][17][20], utilizing normalized coverage as the evaluation metric. As shown in Fig. 6 (a), our SIS scheme outperforms existing methods. Note that our approach is simpler by using only a Grasp&Fling motion, while SpeedFolding needs additional motions such as Pick&Place and Drag in later steps. In addition, the proposed SIS aligns the T-shirt to a specific goal configuration, which SpeedFolding does not consider. We use IoU of resulting configuration with goal configuration to evaluate the orientation alignment performance on Category A. Our proposed SIS reaches 0.799, 0.826, and 0.831 average IoU within three, four, and five episode steps, respectively.

In Fig. 6 (b) and (c), we have also plotted the normalized coverage of Category B and C. Category B contains two short-sleeved T-shirts with a pocket attached to the front with seams. Category C contains two short-sleeved sports T-shirts that have solid seams that are not in the contour area when fully unfolded. According to Welch’s t-test, there is no significant difference in performance between experiments using Categories A and B. Therefore, we conclude that the seams on the pocket do not significantly affect the unfolding performance of our scheme. This is because the priority of selecting the dotted seams corresponding to the pockets is low, as shown in Fig. 4. Even for the challenging cases of Category C, the average coverage in ten trials of our scheme reaches more than 0.83 within five episode steps, demonstrating that our scheme can also be applied to such types of sports T-shirts.

Using the same models without additional training/updating, we conduct experiments on different types of garments, such as long-sleeved T-shirt and sleeveless shirt, shown in Category D and E in Fig. 7. A total of ten unfolding trials are performed on two cloth samples for each type of garment. The results are shown in Fig. 6 (d) and (e). Although the performance suffers some degradation, our scheme can still unfold the garments without any additional retraining/updating. We conclude that our scheme can be applied to these two types of garments. Example results of unfolding trials are shown in Fig. 8.