Precise Affordance Annotation for Egocentric Action Video Datasets

Earlier affordance datasets [15, 12] annotated possible actions and the exact regions where actions could occur for object images. Koppula et al. [9] provide affordance label annotation for human-object interaction video clips. Thermos et al. [18] and Fang et al.[3] annotate human-object interaction hotspot maps as object affordance for video clips associated with their action labels. Furthermore, Nagarajan et al. [13] use the action labels of the EPIC-KITCHENS dataset, which is egocentric, as weak supervision to learn to generate human-object interaction hotspot maps. These datasets neither provide a clear definition of affordance nor consider humans’ motor capacity. Therefore, we propose a precise affordance annotation scheme considering both humans’ motor capacity and the object’s physical property for video datasets.

Affordance understanding methods can be divided into four categories: Affordance Recognition, Affordance Semantic Segmentation, Affordance Hotspots Prediction, and Affordance as Context. Given a set of images/videos, the task of affordance recognition [1] aims to estimate object affordance labels from them. Affordance semantic segmentation [11, 13, 3] aims at segmenting an image / video frame into a set of regions that are labeled with affordance labels. Some works  [10, 14] also use affordance as a context for other tasks such as action anticipating.

All of these methods are influenced by the lack of precise affordance annotation. For example, our attention differs when observing the “cut” action and the “take” action [8, 7]: the former on the interacting object and the latter on the hand. Mixing them up may confuse models for affordance recognition. Moreover, depending on objects, we may perform the same action with different affordances. For example, we may directly push or handle the doorknob to close a door. However, previous works overlook these details by simply using the action ”close” as an affordance label. This leads to the failure to distinguish different affordance hot spots in affordance semantic segmentation tasks.

Our proposed affordance annotation scheme is inspired by the three-action system model (3AS) [16]. The three-action system model includes affordance, mechanical action, and contextual relationship. We mainly focus on the first two concepts since they are closely related with hand-object interaction. Affordances are hand-centered, animal-relative, and goal-irrelevant properties of an object. Mechanical actions are tool-centered, and mechanical action possibilities between objects. Thus, we separately consider the affordance and mechanical action for an instance of hand-object interaction as shown in Figure  1. Affordance labels are defined as a combination of goal-irrelevant motor actions and grasp types, and we use verbs describing actions between objects as mechanical action labels.

To annotate an action video dataset, we first need to divide the original action labels of the dataset into tool-use actions and non-tool-use actions. Then annotate mechanical actions for tool-use actions and affordances for both tool-use and non-tool-use actions.

Tool-use / non-tool-use action annotation: Tool-use / non-tool-use action annotation for action video datasets can be done by dividing original action labels of the dataset into three categories: tool-use action, non-tool-use action, and both, according to the meaning of each action label. For example, “take” is a non-tool-use action, while “cut” is a tool-use action. Some action labels could represent both the tool-use action and the non-tool-use action simultaneously, such as “wash”. We ignore these labels during annotation because of their ambiguity.

Mechanical action annotation: We only need to annotate mechanical actions for tool-use actions. According to the definition, we can use the verbs of original action labels as mechanical action labels. For example: in “stir food”, “stir” is the mechanical action between the slice and the food. We can automatically annotate mechanical actions for all tool-use action video clips based on this rule, allowing significant reduction of annotation cost.

Affordance annotation: For affordance annotation, as shown in Figure 2, we annotate a goal-irrelevant motor action and a grasp type for each video clip. Given an unlabeled video clip, we first define a goal-irrelevant motor action according to the object property used in it. In the example of this figure, we use “pull” to represent the “pullable” property of the cupboard. Next, we chose a grasp type from a 6-class grasp types taxonomy. This taxonomy is simplified from a well-known 33-class grasp type taxonomy [5] based on the power of the grasp type and the posture of the thumb. Finally, we combine the goal-irrelevant motor action label with the grasp-type label as the affordance label. This form of affordance labels can model both the object’s physical property and the human motor capacity.

To reduce the manual work of affordance annotation, we propose an efficient annotation method. As shown in Figure 3, there are multiple video clips contain the same participant performing the same action on the same object in the original dataset. We first sample five video clips from video clips with the same verb (action)-noun (object)-participant annotation, then manually annotate affordances for them. Finally, we use these affordance labels as the affordance annotation for all video clips with the same verb (action)-noun (object)-participant annotation. The efficiency and accuracy of this method are shown in Section 4.1.

We annotated the EPIC-KITCHENS dataset with our proposed method. Inside the verb list of the original dataset, we select 33 verbs as mechanical action annotations. This results in 8.5k tool-use action video clips with mechanical action annotations. There are 51.5k non-tool-use action video clips in total. Then, using our annotation method, we sampled and annotated a total of 300 verb-noun pairs, obtaining 31,924 video clips and 24 affordance labels. For quality check, we randomly sample 1,000 instances from the annotated video clips and manually checked their affordance label, getting an accuracy of 96.76%.

To validate the rationality of our annotation, We compare the performance of a model trained with random annotation and a model trained with our annotation on the EPIC-KITCHENS dataset. We use a slowfast [4] model as the classifier. The result of tool-use/non-tool-use action classification is shown in Table 1. From the table, the model trained and tested with our annotation performs significantly better, while the model trained and tested on the random annotation can only get chance level performance.

We benchmark the tasks of mechanical action recognition and affordance recognition. We also use slowfast for these two tasks. For the 33-class mechanical action recognition, we get a recognition accuracy of 51.90%. For the 24-class affordance recognition task we obtain a recognition accuracy of 31.07%.

The visualization results generated by GradCam [17] are shown in Figure  4. From the first row, we can see that the affordance recognition model focuses more on hands than on objects. The second row shows that the mechanical action recognition model cares more about the interaction between objects. When the second row is compared to the third row, it is evident that the mechanical action recognition model focuses on tool-object interactions (2nd row), and the tool-use/nontool-use action classification model focuses on the existence of tools (3rd row).