Interacted Object Grounding in Spatio-Temporal Human-Object Interactions

To support practical ST-HOI learning, we collect third-view videos from large-scale dataset AVA (Gu et al. 2018). It contains 430 videos with spatio-temporal labels of 80 atomic actions (body motions and HOIs). As AVA includes complex HOIs in diverse scenes, it can bring great visual diversity to our benchmark. We extract the HOI-related frames and the corresponding human boxes and action labels, thus the clips in GIO have uneven temporal durations. Notably, we only consider the non-human objectives in HOIs. Overall, based on the available train and validation (val) sets of AVA 2.2 (Gu et al. 2018) (299 videos), we chose 74 hours of video including 51 actions (detailed in the supplementary).

AVA provides labels with a stride of 1s, so we add boxes and class labels for all interacted objects with the same stride. Following AVA, we define the annotated frame as key frames which are at 1-second intervals.

First, as humans can perform multi-interaction simultaneously, we set the annotating unit as a clip including one single interaction to normalize the annotation. For example, a 30s clip including an actor holds-sth (1-30s) and inspects-sth (10-15s), will be divided into two sub-clips, i.e., a 30s sub-clip for holds-sth and a 5s sub-clip for inspects-sth. In brief, each sub-clip contains one verb and one/several class-agnostic interacted objects. Then, sub-clips are annotated separately, and each one is annotated by at least 3 annotators and checked by an expert to ensure quality.

Second, as AVA contains various scenarios and diverse objects, to better locate objects and avoid ambiguity, each annotator is given a whole sub-clip to draw boxes and classify them. In default, we use COCO (Lin et al. 2014) 80 objects as a class pool. If annotators think an object is not in the pool, they are asked to input a suitable class according to their judgments. If an object cannot be recognized, they can choose the “unknown” option. Then, we find that a surprising 42.66% of object instances are beyond our pool. After exhaustive annotation, we fix the input typos, exclude outliers via clustering, and combine similar items. Finally, 1,098 classes are extracted after cleaning. We then conduct re-recognition for the frames including “unknown” objects.

Finally, to generate the ST-HOI labels, we further consider the objects in each sub-clip (one interaction of one person). If there is only one object in a sub-clip, we use its locations as the labels. If there are multiple objects, we record all of their boxes and manually link their boxes as multiple-object tracklets. Then, each sub-clip is seen as a ST-HOI traklet, whose label records a human actor tracklet, an interaction, a/several class-agnostic object tracklets.

GIO includes 290K HOI triplets and 290K object boxes of 1,098 classes, including a wide range of rare objects. Only 20.85% of our object classes are covered by the recent large-scale object dataset FSOD (Fan et al. 2020). It is noteworthy that Action Genome and VidHOI include predicates such as next to, which are not HOIs. Consequently, we refined the annotations and recalculated the statistics in Tab. 1. In contrast, GIO, aiming for diversity and finer granularity, offers the highest number of object classes and the richest HOI instances per frame (2.30).

GIO supports ST-HOI detection and many fine-grained tasks, like object classification. However, the original ST-HOI task, involving detection, tracking, and action recognition, is highly complex and challenging, with most approaches facing significant difficulties due to the task’s inherent complexity and the quality of annotations. So instead of requiring vision systems to detect complete ST-HOI triplets, we focus on GIO’s capability for interacted object grounding, i.e., given the human actor tracklet (and the interaction semantics), while object labels are not included in the interaction semantics, probing the ST-HOI understanding from the object view. To make our task realistic, 328 object classes only have less than 5 samples (boxes) in our train set, and 98 classes are unseen in the inference.

Given a clip $C$, the target human tracklet $T_{h}=\{I^{k}_{h}\}^{n}_{k=1}$ ($n$ for tracklet length), we aim at learning a model $\mathcal{M}$ as

where $\hat{T_{o}}$ is the predicted interacted object tracklet and the interaction semantics $s$ is an optional input to inform the system with high-level semantics. To achieve this, instead of directly regressing the object box, we adopt a novel 4D question-answering paradigm to leverage HOI prior. Given the strong generalization ability of SAM (Kirillov et al. 2023a), we adopt it as an objectness detector to generate candidate object proposals (Sec. 4.2). The clip $C$ is first fed to SAM, resulting in $K$ candidate object mask tracklets $M_{o}=\{M_{o}^{i}\}_{i=1}^{K}$. The task is then reformulated as choosing the interacted mask tracklets from the candidate tracklets, as

To tackle the challenging GIO, a 4D question-answering network is devised as shown in Fig. 2. Multimodal features, including 4D clues, are extracted in the inspiration of DJ-RN (Li et al. 2020a). We begin by extracting spatiotemporal features from the video using the SlowFast (Feichtenhofer et al. 2019) network as a basis. Then, the 4D Human-Object layout is reconstructed for feature extraction (Sec. 4.3). Finally, we ground the interacted object with two decoders to summarize the important clues in complex spatiotemporal patterns (Sec. 4.4). Despite the suboptimal precision of 4D Human-Object reconstruction, it is effective in alleviating the view ambiguity in clips, also enhancing the object localization with 3D spatial information. The question-answering paradigm eases the learning process.

We chose SAM as the candidate proposal generator for several reasons. First, SAM, based on pixel-level segmentation, provides a finer granularity and more accurate segmentation. Second, AVA consists of many video scenes that are dark, complex, and contain numerous objects. Traditional detection methods struggle to accurately predict small and blurry objects in such challenging scenarios. In contrast, SAM’s pixel-based segmentation is more robust and accurate than directly predicting object bounding boxes. In addition, SAM is also adept at dealing with large objects. However, SAM could segment objects into multiple parts. Thus, our policy is to predict vast majority of the masks belong to the object resulting in a highly accurate bounding box.

Mask proposal generation. Given a clip $C$, we denote the keyframe as $C_{k}$. SAM is first fed with a grid of point prompts on $C_{k}$. Then, low-quality and duplicate masks are filtered out. As a result, each image would produce at most 255 masks as $M_{o}$, which will be sent to the model as proposals to generate the final object box.

GT proposals. To judge which mask is GT, we input the GT object box to SAM as the prompt to get an accurate mask ($M_{acc}$).

Next, we calculate the area of the intersection between the proposal masks and the accurate mask ($A_{inter}$), and divide them by the area of the proposal masks $A_{p}$ to get a ratio for each proposal mask as $ratio=A_{inter}/A_{o}$. Masks with a ratio greater than 0.9 are identified as GT masks. Fig. 3 demonstrates the above process.

To fully leverage the temporal and spatial continuity features of videos, including object information, HOI details, and spatial relations from multiple views, we employed a multi-modal feature extraction approach.

Context Feature. We utilize widely-used SlowFast (Feichtenhofer et al. 2019) to extract context features from the video clip $C$. The features from slow and fast branches are pooled along the time axis, then concatenated into the context feature map $f_{c}\in\mathcal{R}^{H\times W\times D}$, with $H\times W$ being the feature map resolution, and $D$ is the feature dim.

Object Feature. We first resize the masks $M_{o}^{i}$ of the $i$-th mask concerning the context feature map, then the feature for the $i$-th mask could be computed as

where $\otimes$ indicates element-wise multiplication. The object feature is denoted as $f_{o}\in\mathcal{R}^{N_{o}\times D}$ with $N_{o}$ masks.

Language Interaction Feature is optional. If adopted, we input the language-guided query embedding $f_{v}\in\mathcal{R}^{D}$ of GroundingDINO (Liu et al. 2023), which needs a language prompt and the key frame as input. Some other interaction features are discussed in Sec. 5.6.

4D Human-Object Feature. Inspired by Li et al. (2020a), which utilizes 3D information for HOI learning, we incorporate 3D information into our pipeline to exploit the rich HOI prior carried by 4D information. Specifically, we lift GIO to 4D by reconstructing the HOIs in 3D. However, lifting GIO to 4D is challenging given its diverse objects. Existing efforts usually require 3D templates for the objects, which is inapplicable for open-world GIO. To alleviate this, we adopt depth estimation for holistic scene estimation, bypassing the need for object templates. Then, we align the human and scene for consistent 4D H-O representation. Finally, we extract the 3D feature with the lightweight base point set (BPS) (Prokudin, Lassner, and Romero 2019).

1) Human reconstruction. Considering that videos without scene switching allow for better human tracking and less processing time of 3D data, we first perform shot detection and segment the original video into multiple sub-clips. Then, PHALP (Rajasegaran et al. 2022) is adopted to recover 4D human tracklets from the sub-clips in SMPL (Loper et al. 2015) representation. The 3D humans are further represented as SMPL mesh point clouds $p_{h}\in\mathcal{R}^{T\times N_{h}\times V\times 3}$, where $T$ is the length of the clip, $N_{h}$ is the number of existing human instances, and $V$ is the number of mesh vertices.

2) Scene reconstruction via depth estimation. We use ZoeDepth (Bhat et al. 2023) to estimate the depth of the corresponding clip and transform them into scene point cloud $p_{s}\in\mathcal{R}^{T\times N_{p}\times 3}$, where $N_{p}$ is the number of points.

3) Human-Scene alignment. The humans and scenes are initially inconsistent in scale and position. To align them, we render the $N_{f}$ front surface vertices $p_{h}^{f}\in\mathcal{R}^{N_{f}\times 3}$ of the human mesh to the image space, find the corresponding pixel of each vertice, and locate the corresponding point in the scene point cloud $p_{s}^{f}\in\mathcal{R}^{N_{f}\times 3}$. Next, we align $p_{h}^{f}$ and $p_{s}^{f}$ by calculating the scale and displacement of $p_{s}^{f}$ to align with $p_{h}^{f}$. We calculate scale $s$ and displacement $b$ as

In detail, the scale is calculated as the ratio between the average pairwise distance of $p_{h}^{f}$ and $p_{s}^{f}$, while the displacement is calculated as the displace between the center point of $p_{h}^{f}$ and $p_{s}^{f}$. The aligned human-scene point cloud is then formulated as $p=(p_{h},p_{s}\cdot s+b)\in\mathcal{R}^{T\times(N_{h}\times V+N_{h}\times N_{p})\times 3}$.

4) 3D feature extraction. We adopt BPS to extract features, which is simple and efficient for encoding 3D point clouds into fixed-length representations. We randomly select $\frac{D}{2}$ fixed points in a sphere and compute vectors from these basis points to the nearest points in a point cloud; then use these vectors (or simply their norms) as features, shown in Fig. 2. We adopt the human pelvis joint as the sphere center for base point generation. We selected a radius of 1.5 times the height of the human body to cover the range of human interactions. In this way, in one space, we obtain $T\times N_{h}\times\frac{D}{2}$ base points. We calculate the distances from these base points to the human mesh point cloud and the scene point cloud, treating them as features. Then we concatenate human features and scene features to get the final 3D feature $f_{3D}\in\mathcal{R}^{(T\times N_{h})\times D}$, in the following we refer to $T\times N_{h}$ by $N_{3D}$, i.e., $\mathcal{R}^{N_{3D}\times D}$.

We utilize a 2D transformer decoder and a 3D transformer decoder to integrate multi-modal features. The 2D decoder outcome is sent to the 3D decoder as a query via an MLP as the 3D adapter. Note that the 2D decoder results have already been satisfactory, but the 3D decoder could further enhance predictions from the 3D perspective. Each 2D decoder query $Q_{s}\in\mathcal{R}^{N_{q}\times D}$, is obtained via $Q_{s}=Q_{v}+Q_{h}$, where $Q_{v}\in\mathcal{R}^{N_{q}\times D}$ is the optional verb semantic query from the feature vector $f_{v}$, and the human query $Q_{h}\in\mathcal{R}^{N_{q}\times D}$ is obtained via a temporal pooling, a ROIAlign pooling, and a spatial pooling of the SlowFast features with the human bounding box. Given the context feature $f_{c}$, the object feature $f_{o}$, we concatenate them as the key and value of the 2D decoder. The object feature $f_{o}$ and the 3D feature $f_{3D}$ are concatenated as the key and value of the 3D decoder.

The 2D/3D decoder outputs feature $f_{q}$. The cosine similarity between $f_{q}$ and all object mask features $f_{o}$ is computed. Then, we derive scores for each query relative to each mask, denoted as $S_{m}^{i}$. Higher scores suggest a greater likelihood of the mask being associated with the target object. Considering that a person tends to interact with objects that are closer in proximity, we use the distance between masks and humans to assist us in calculating mask scores. The distance of the $i$-th mask is computed as $S_{d}^{i}=dist(C_{h},C_{m}^{i})$, where $C_{h}$ and $C_{m}^{i}$ refer to the human box and the $i$-th mask’s box. Ultimately we adopt the GIoU (Rezatofighi et al. 2019) distance. The final score of the $i$-th mask is computed as

where $\gamma$ is a weight. Then, we introduce a threshold $\tau$ to determine whether a mask is considered part of the target object. In the results for a certain query, if none of the mask scores exceed this threshold, we select the mask with the highest score. We cluster the predicted masks based on their depths and then determine the boundaries(detailed in the supplementary material). For a given object w.r.t. $i$-th mask, BCE loss $L_{o}^{i}$ is used for supervision. The overall loss is computed as $L_{o}=({\textstyle\sum_{i=1}^{N_{o}}L_{o}^{i}})/N_{o}$.

Modified versions of mean Average Precision (mAP) and mean Intersection over Union (mIoU) are adopted. For each GT tracklet, we sort all predictions by their scores in descending order. We identify the first prediction with an IoU higher than a threshold as a hit and calculate its precision by its position in that order. mAP is averaged across all test instances. For mIoU, we calculate all IoUs between the GT and predicted boxes and report the largest IoU. To take into account the precision of the prediction, a weighted mIoU is proposed as $\rm{mIoU_{w}}$. For each GT tracklet, predictions are sorted by scores in descending order. The rank of each prediction is used to calculate $\rm{mIoU_{w}}$ as

where $\hat{T_{o}}$ and $T_{o}$ denotes predicted and GT tracklets. Since $\rm{mIoU_{w}}$ is a more reasonable metric, we adopt $\rm{mIoU_{w}}$ instead of mIoU in the experiments.

For the 3D feature, considering the reconstruction quality of the 4D HOI layout, the reconstruction is only conducted for frames with object labels. After filtering, there are 107,663 of 126,700 key-frames attached with 4D HOI layout (85,370 for training, 22,293 for inference). SlowFast pre-trained on AVA 2.2 is adopted for video feature extraction. An Adam optimizer, an initial learning rate of 1e-3, a cosine learning rate schedule, and a batch size of 16 are adopted. When training the 2D decoder, the learning rate of the parameters of SlowFast and Grounding DINO is 1e-5 and the 3D decoder is omitted. When training the 3D decoder, other parts except the 3D decoder are frozen. $N_{3D}$ is set to 256, $N_{o}$ is set to 256 and $N_{q}$ is set to 24 for alignment. Considering that the ground truth mask for each keyframe is sparse, we use weighted BCE loss, where the loss coefficient for true positions is ten times that of false positions.

Our dataset supports different settings for further investigation, like inputting the interaction semantics to the grounding model, using more advanced LLM-extracted features, and the effect on the grounding of different human trackers, etc. However, to focus on evaluating the grounding itself, we mainly discuss the default setting given the interaction semantics and GT human tracklets. For example, our system still predicts the interacted object well, without inputting the language interaction feature or the detected actions and humans from standard SOTA action detectors (Feichtenhofer et al. 2019; Wang et al. 2023). The proposed 4D-QA model has 246M parameters and achieves an inference speed of 8.63 FPS with a batch size of 1 (8 adjacent frames and 1 keyframe) on a single NVIDIA 3090 GPU.

We adopt six models of four different types as our baseline. It is worth mentioning that since our task is new, we find these models most close to our task. But, they still do not fit our task very well in the setting. We devise corresponding protocols to adapt these models to our task.

Image/Video-based HOI models. PViC (Zhang et al. 2023) and Gaze (Ni et al. 2023) are adopted as conventional image/video-based HOI detection baselines. Given a frame or clip and a human bounding box $b$, the HOI models input the frame or clip and output a series of HOI triplets as $\langle b_{h},b_{o},p\rangle$, where $b_{h},b_{o}$ are human and object bounding boxes, and $p$ is the predicted interaction probability. We preserve all the results with $IoU(b,b_{h})>0.5,p>0.2$, and the corresponding $b_{o}$ are adopted as the grounded objects.

Open-vocabulary object detection models. Detic (Zhou et al. 2022) is adopted, inputting a frame and expected object categories, outputting $\langle b_{o},p\rangle$ as object bounding boxes and objectness score. Results with $p>0.5$ are preserved and paired with the human query as the grounded objects.

Visual grounding models. Grounding DINO (Liu et al. 2023) is adopted, which takes a frame and a text prompt $s$ as input and produces $\langle b_{o},p\rangle$ as grounded box and confidence. We also test a video-grounding baseline CG-STVG (Gu et al. 2024), which aims to predict a spatial-temporal tube for a specific target subject/object given some semantic $s$. $s$ is in the format as “The object that the person is {interacting with}”, where the placeholder “{Interacting with}” could be replaced with a specific action name. All the outputs are paired with the human query as the grounded object.

LLM based models. Qwen-VL (Bai et al. 2023) is adopted. It takes a frame and the text prompt “Output the bounding box that the person is {interacting with}.” and produces the bounding box $b_{o}$ if detected.

Results are shown in Tab. 2. For all the models, we combine the human-object distance for $\rm{mIoU_{w}}$ and mAP as Eq. 5. All baselines provide sub-optimal performance, indicating their deficiency in interactiveness grounding. Also, most baselines take little use of temporal information since they utilize only images as inputs, leading to bad performances on “temporally hidden objects” such as the chairs obstructed by humans sitting on them but appearing in the next frame.

As HOI detection models, PViC and Gaze fail to perform well due to the large number of novel objects in GIO. The open-vocabulary object detection model Detic demonstrates low $mIoU_{w}$ since it cannot discriminate the interacted objects related to humans (interactiveness (Li et al. 2019b)). It is noteworthy that Detic tends to predict a substantial number of object bounding boxes, sometimes exceeding 900, with many false positive predictions. CG-STVG, lacking pre-training on large datasets and integration of visual-language models, outperforms PViC, Gaze, and Detic in mIoU_w, using a single high-quality bounding box per HOI instance for higher mIoU_w despite lower mAP. Grounding DINO performs better than other baselines, but it is still limited for “hidden objects”. Also, it frequently fails to fully understand the interaction semantics. Qwen-VL, a large vision language model, provides decent $mIoU_{w}$ but poor $mAP$s, which suggests that although Qwen-VL can localize the approximate positions of most objects, it struggles to detect precise bounding boxes. Our model performs well in localizing diverse and unseen objects, where the baselines struggle. Also, our model demonstrated decent $mIoU_{w}$. These experimental findings indicate that our method excels in object grounding for spatiotemporal HOI understanding.

In addition, we considered ST-HOI as the task design, resulting in the highest mAP of 6.8, i.e., the ST-HOI task is kind of too challenging even ignoring the annotation missing problem. This demonstrates the rationality and exploratory potential of the GIO task formulation.

We also evaluate 4D-QA on VidHOI (Chiou et al. 2021) with the GIO task. The GIO-pretrained 4D-QA gets 14.23 mAP and 22.66 mIoU_w and 25.35 mAP and 29.61 mIoU_w after 10 epoch finetuning. GroundingDINO gets 15.87 mAP and 17.57 mIoU_w. Results on VidHOI reveal that grounding interacted objects is challenging enough to be carefully explored and our 4D-QA maintains decent performance. VidHOI only has 1.22 HOIs/frame in the filtered (Sect. 3.3) test set, allowing GroundingDINO to slightly outperform zero-shot 4D-QA on mAP because Grounding DINO performs better to localize the only object in the one-HOI frame.

Fig. 4 visualizes the grounded interacted object in 3 consecutive frames. The predicted masks (colored regions) are integrated into final object boxes (green) as in Sec. 4.4.

We conduct ablation studies on the different components of our model on the GIO test set as reported in Tab. 3.

Distance. Removing the use of distance would result in a degradation (22.07 mAP, 27.85 mIoU_w). Replacing the GIoU distance with the L2 distance would also cause a decline in performance (22.34 mAP, 28.39 mIoU_w).

3D Feature $f_{3D}$. 4D-QA w/o $f_{3D}$ presents a performance decrease (22.67 mAP, 28.76 mIoU_w). Fig. 5(a) further shows the influence of $f_{3D}$ w.r.t. different data characteristics, namely the relative object size and H-O distance. As shown, our methods provide superior performance compared to Grounding DINO across different data groups, especially on medium-to-large objects and close-to-medium H-O pairs. For 2D and 3D’s difference, $f_{3D}$ gains 4.68 mIoU_w on large objects and 1.32 mIoU_w/1.59 mIoU_w on close/far H-O pairs. Besides, we find $f_{3D}$ outperforms 2D by 2.25 mIoU_w and 2.83 mAP@0.5 on 40 verb classes, especially on verbs like drive, exit, press that involve large or occluded objects or occur in complex scenarios. Notably, in Fig. 5(b) the relative improvement that $f_{3D}$ brings increases with the IoU threshold requirement for mAP, indicating that $f_{3D}$ contains more accurate predictions than 2D.

Interaction Feature. We replaced the Grounding DINO interaction feature with the Bert and CLIP interaction language embedding. In addition, we performed a test without the interaction feature. As shown, the sophisticated feature from the Grounding DINO can help the grounding task to better utilize the interaction information, while the simple language representation difference between Bert and CLIP affects the performance little. Eliminating the interaction feature brings a major performance degradation.

Predicted human boxes, with 88 mIoU w.r.t. GT human boxes, were used as human queries. The slight performance drop indicates the robustness and flexibility of 4D-QA.

Box Regression. We used an MLP after the decoder to directly regress the boxes instead of utilizing SAM mask candidates or other box proposals. The performance drop shows the importance of SAM-generated mask candidates.