Act the Part:
Learning Interaction Strategies for Articulated Object Part Discovery

General Setting. Let $\mathcal{O}$ denote a set of articulated objects, each with $n\leq N$ parts. At each timestep $t$, an agent gets an observation $I_{t}\in\mathbb{R}^{H\times W\times C}$, and executes an action $a_{t}\in\mathcal{A}$ on an object $o\in\mathcal{O}$, where $\mathcal{A}$ is the set of all possible actions. Additional sensor readings $s_{t}\in\mathbb{R}^{l}$ complement visual perception. The action results in the next observation $I_{t+1}$. Given the sequence of $T$ observations, sensor readings, and actions, the goal is to infer part mask $\mathcal{M}_{T}\in\{1,2,...,N+1\}^{H\times W}$, where each pixel is assigned a value corresponding to $N$ part labels or background.

Task Details. We consider $\mathcal{O}$ to be a set of common household objects with $n\leq 3$ parts, $T=5$, $W,H=90$, and $C=3$ (RGB). All objects have revolute joints and no fixed base link. Each $a\in\mathcal{A}$ represents a tuple: an image pixel to hold, another pixel to push, and one of eight push directions. The directions are discretized every 45$\degree$ and are parallel to the ground plane. We take $s_{t}\in\{0,1\}^{3}$, representing binary signals for detecting contact on the hold and push grippers and a binary sheer force reading on the hold gripper to emulate touch.

Environment Details. To enable large-scale training and ground truth part segmentation (for benchmarking only), we use a simulated environment. However, we also show our model generalizes to real-world images without fine-tuning. Our simulation environment is built using PyBullet [11] with Partnet-Mobility [9, 29, 49] style dataset assets.

Conditional Bimanual Action Inference. The interaction task reduces to finding pixels to hold and push and determining the push direction. To decrease the search space, we discretize the push direction into eight options (45$\degree$ apart). We consider a constant magnitude push force parallel to the ground plane. We condition the predicted push location and direction on the predicted hold location. This allows us to synchronize sub-actions without considering all pairs.

Interaction Network. At every timestep, we predict one step pixel-wise reward for holding and pushing at the spatial resolution of the input image, similar to Zeng et al. [56]. As shown in Fig. 3, we use a shared ResNet18 [16] with two residual decoder heads wired with U-Net [39] skip connections. At each timestep $t$, we have a current belief about the part segmentation. This is represented as a part memory $V_{t}\in\{0,1\}^{H\times W\times N}$, where each channel encodes a different part mask. Given an image $I_{t}$ and $V_{t}$, the network predicts a hold reward map $H_{t}\in[0,1]^{H\times W}$, where each entry estimates the reward for holding that pixel. We uniformly sample one of the top $k=100$ pixels from $H_{t}$ as the hold location. Sampling encourages optimization over the top $k$ actions, which we notice is necessary for the model to learn effective strategies.

Part Network. Our part network (Fig. 4) takes the observations before and after the interaction. Additionally we pass in the hold location $h_{t}$ and a spatial encoding $p_{t}$ of the push location and direction. $p_{t}$ has a 2D Gaussian centered at the push pixel, analogous to $h_{t}$. To encode direction, we add Gaussians of smaller mean value in the direction of the push, forming a trail. The network is comprised of a shared encoder with two decoder heads to predict $M_{t}$ and $M_{t+1}$. Using consistent forward and backward flow collected during interaction network training, we threshold at zero to acquire target motion masks. We supervise predictions using binary cross-entropy loss.

New Part. If $M_{t}$ does not overlap significantly with any channels in $V$, it is likely to be a new part. A free channel $c$ is assigned: $V^{c}\leftarrow M_{t+1}$. If there is significant overlap between $M_{t}$ and a mask $V^{c}$, relative only to the area of $M_{t}$, there is indication two parts are assigned to $V^{c}$ that must be split: $V^{c}\leftarrow V^{c}-(V^{c}\cap M_{t})$ and $V^{c+1}\leftarrow M_{t+1}$. Finding a new part is the most desirable case.

Existing Part. If there is significant overlap between $M_{t}$ and a mask $V^{c}$, relative to the areas of both $M_{t}$ and $V^{c}$, we execute the update: $V^{c}\leftarrow M_{t+1}$. This case is less desirable than discovering a new part.

Entangled Parts. If there is significant overlap between $M_{t}$ and a mask $V^{c}$, relative to the area of only $V^{c}$, it suggests our action is entangling movement of more than one part. During training; $V^{c}\leftarrow M_{t+1}$. During testing, we use Iterative Closest Point (ICP) to get the correspondences between $V^{c}$ and $M_{t+1}$, yielding $T\in SE(2)$, to execute the updates: $V^{c}\leftarrow(T\circ V^{c})\cap M_{t+1}$, then $V^{c+1}\leftarrow M_{t+1}-V^{c}$. Entangled part actions are the least desirable, as reflected in our reward described next.

Average Percentage Error (APE). To measures errors in number of parts discovered, we compute $|(|\mathcal{G}|-|\mathcal{H}|)|/|\mathcal{G}|$.

Part-aware Intersection over Union (IoU). We use Hungarian matching to solve for the maximal IoU bipartite match between $\mathcal{G}$ and $\mathcal{H}$. Unmatched parts get IoU of 0. Final IoU is determined by summing part IoUs and dividing by $\max(|\mathcal{G}|,|\mathcal{H}|)$. The metric penalizes both errors in mask prediction and failure to discover masks (e.g. if one of two parts is discovered, maximum IoU is 50%).

Part-aware Hausdorff distance @ 95% ($\mathbf{d_{H95}}$). We notice IoU is sensitive for thin structures. For example, a small pixel shift in a thin rod can lead to IoU of 0. To provide a better metric for these structures, we measure $d_{H95}$, which is a part-aware variant of a common metric in medical image segmentation [8]. The directed Hausdorff distance @ 95% between some masks $G\in\mathcal{G}$ and $H\in\mathcal{H}$ is defined as $d^{d}_{H95}(G,H)=\underset{g\in G}{P_{95}}\min_{h\in H}||g-h||_{2}$ where $P_{95}$ gives the 95-th percentile value over pixel distances. The metric is robust to a small number of outliers, which would otherwise dominate. The symmetric measure is given as: $d_{H95}(G,H)=\max(d^{d}_{H95}(G,H),d^{d}_{H95}(H,G))$. We use Hungarian matching to find minimal $d_{H95}$ bipartite matches between $\mathcal{G}$ and $\mathcal{H}$. If $|\mathcal{G}|\neq|\mathcal{H}|$, we compute the distance of unmatched parts against a matrix of ones at the image resolution. Distances are summed and normalized by $\max(|\mathcal{G}|,|\mathcal{H}|)$.

Effective Steps. A step is effective if the hold is on an object link, the push is on another link, and the action creates motion.

Optimal Steps. An interaction is optimal if it is effective and a new part is discovered. If all the parts have already been discovered, moving a single existing part in the interaction is not penalized.

Baselines and Ablations. We compare the AtP framework trained with and without touch reward, [Ours-Touch] and [Ours-NoTouch] respectively, with the following alternative approaches to study the efficacy of our interaction network. All methods use the same part network trained from the full AtP rollouts:

• •

  Act-Random: hold and push locations and the push direction are uniformly sampled from the action space.

• •

  Act-NoHold: The agent applies a single push action every step. Single modes of interaction are widely used in interactive segmentation algorithms [13, 5, 46, 32, 33, 12]; however, this ablation differs from these works as push is learned.

• •

  Act-NoPart: The interaction network does not take the part memory and considers each moved part as a new part for reward calculation.

For the modified reward used to train the above networks see Appx. C. We also design two oracle algorithms using simulation state to provide performance upper bounds:

• •

  GT-Act: Optimal action based on ground truth state, but use of AtP part network for the mask inference. This is conceptually similar to [34], which uses expert actions for part segmentation.

• •

  GT-Act-Mot: Optimal action based on ground truth state with motion masks from the ground truth flow.

Does Interaction Help Part Discovery? First we want to validate if AtP learns effective interaction strategies for part discovery by accumulating information over time. To evaluate, we plot the part mIoU w.r.t. interaction steps in Fig. 6. As expected, the upper bounds peaks at 2 and 3 steps for pliers and multilink, respectively. While other algorithms’ performance saturate quickly with one or two interactions, [Ours-Touch] and [Ours-NoTouch] are able to improve with more interactions. These plots indicate that while the learned interaction strategies may not be optimal (compared to upper bounds using ground truth state), they are informative for discovering new parts of the object and self-correct errors over time. Results from other categories are presented in Appx. G, where we see all AtP curves approach the upper bounds.

Does Part Prediction Help with Action Selection? Our interaction network takes the current belief of the part segmentation as input and obtains reward for new part discovery. We hope this design would encourage the algorithm to focus on selecting actions that provide information gain (e.g., push new parts to discover them). To validate this design, we compare AtP to an ablated version, [Act-NoPart], which is not mask-aware. Interestingly, this model performs efficient actions at roughly the same rate as [Ours-Touch] (Fig. 7); however, [Ours-Touch] is better at finding optimal actions (resulting in new part discovery). Histograms for all other categories are presented in Appx. G and corroborate these findings. This result is also supported in Tab. 2, which shows degradation on all perceptual metrics when part-awareness is not exploited.

Is Holding Necessary? In contrast to a vast majority of prior work that use simple push actions, our algorithm uses bimanual actions for object interaction (i.e., simultaneous hold and push). Our hypothesis is that such actions give the system a better chance at disentangling motion between different moving parts and therefore aid part discovery. To validate this hypothesis, we compare our algorithm with an agent that performs only push actions [Act-NoHold]. The result in Tab. 2 shows that without the hold action the system performance is much worse at part segmentation. [Act-NoHold] has has trouble discovering more than one object part, since the whole object is likely to be moved during interaction. Furthermore, this result suggests more complex perceptual modules are necessary to get push-only policies to achieve competitive performance at this task. While this is an interesting direction, disentangling the motion of many moving parts is non-trivial and out of scope for this paper.

Does Touch Feedback Help? In this experiment, we want to evaluate the effect of touch feedback. Looking at Tab. 2, we see that [Ours-Touch] outperforms [Ours-NoTouch] in most categories. A similar trend is noticeable when looking at action performance in Figs. 6 and 7. We conjecture this is due to the benefit of using touch signal to define more specific reward cases and to make reward more dense, which is ultimately reflected in the final system performance. However, we are still able to learn helpful interaction strategies even without touch.

Generalization to Unseen Objects and Categories. Our algorithm does not make category-level assumptions, therefore the same policy and perception model should work for unseen object categories with different kinematic structures. More specifically, we wish to probe generalization capabilities of our model to unseen instances from seen categories and novel categories.

Conditional Action Reasoning. We visualize the conditional action inference result from the interaction network on real world images. Fig. 8 shows two types of visualizations. In example (a), we pick various hold positions and analyze the “push right” reward prediction maps (recall: pushing is conditioned on holding). We notice that the affordance prediction switches between the links depending on the hold location, which indicates the network’s understanding about the object structure. When hold is placed in free space or between the links, the push reward predictions are not confident about pushing anywhere. These results suggest that our model is able to disentangle push predictions from its own hold predictions, thereby demonstrating a form of conditional reasoning.

Interaction Experiment. Next, we evaluate both perception and interaction networks together with the real world physical interactions. To validate performance independent of robot execution accuracy, a human is instructed to execute the actions. Fig. 9 shows the predicted actions, affordances and final object part masks discovered by the algorithm. Without any fine-tuning, the algorithm shows promising results on inferring interaction strategies and reasoning about the observed motion for part discovery. Please refer to Appx. G for more real world experiment results and failure case analysis. Note that no quantitative analysis can be made for this experiment as there is no ground truth part segmentation for these images.