Spatial-Temporal Knowledge-Embedded Transformer for Video Scene Graph Generation

Over the past decade, scene graph generation (SGG) [27, 28, 29, 30] has attracted considerable interest across various communities due to its ability to precisely present the semantics of visual contents of complex visual scenes. It aims to identify all objects and their visual relationships within an image, necessitating visual perception and natural language understanding. Hence, much effort has been invested in aligning visual and semantic spaces for relationship representation learning [31, 32, 33]. Further research has highlighted the importance of each subject-object pair for inferring ambiguous relationships. Xu et al. [1] propose to iteratively refine predictions by passing contextual messages, and Zellers et al. [34] capture a global context using a bidirectional LSTM. Their impressive performance demonstrates that spatial context is crucial for recognizing visual relationships. Therefore, subsequent research underlines the role of spatial context in generating scene graphs. To this end, many works adopt graph convolutional networks [35] or similar architectures to pass messages among different objects. Chen et al. [36] built a graph that associates detected objects according to these statistical correlations and employs it to learn the context among different objects to regularize prediction. Tang et al. [37] design a dynamic tree structure to encode the context among different object regions efficiently. With the impressive progress of transformer [38], more and more works propose utilizing this kind of model to learn more representative features from spatial context. Cong et al. [39] introduce an encoder-decoder architecture to reason about the visual feature context and visual relationships, using different types of attention mechanisms with coupled subject and object queries. Kundu et al. [40] propose contextualized relational reasoning using a two-stage transformer-based architecture for effective reasoning over cluttered, complex semantic structures. Despite impressive progress on static images, leading algorithms may suffer significant performance drops when applied to recognize dynamic visual relationships in videos because it requires an in-depth exploration of temporal consistency/transition correlations across different images.

Building on successfully exploring the spatial context within images, researchers have explored the spatial context and temporal correlation simultaneously in video scene graph generation. With the advent of ImageNet-VidVRD [21], a benchmark of video visual relation detection, numerous approaches [22, 41, 42, 43, 23] have been proposed to employ object-tracking mechanisms to dive into temporal correlations among different image frames. Qian et al. [22] propose to use a graph convolution network to pass messages and conduct reasoning in the fully-connected spatial-temporal graphs. Similarly, Tsai et al. [41] design a gated spatiotemporal energy graph that exploits the statistical dependency between relational entities spatially and temporally. Recently, Teng et al. [43] propose a new detect-to-track paradigm by decoupling the context modeling for relation prediction from the complicated low-level entity tracking. Zheng et al. [23] propose a unified one-stage model that exploits static queries and recurrent queries to enable efficient object pair tracking with spatiotemporal contexts. However, introducing object-tracking models results in high computational cost and memory consumption and quickly overwhelms the valuable information due to many irrelevant frames, leading to sub-optimal performance.

An alternative line of research tries to address the task of video scene graph generation based on detected object proposals. Compared with tracking-based VidSGG algorithms, this kind of method focuses on modeling the temporal context in the narrow sliding window, avoiding the prediction shift resulting from the inconsistency among tracking proposals. Recently, Cong et al. [25] propose a strong baseline that adopts a spatial encoder and a temporal decoder to extract implicitly spatial-temporal contexts. This work demonstrates that the temporal correlation is crucial for inferring dynamic visual relationships. Li et al. [24] propose a novel anticipatory pre-training paradigm to model the temporal correlation implicitly. Wang et al. [44] propose to explore temporal continuity by extracting the entire co-occurrence patterns. Kumar et al. [45] further propose to spatially and temporally localize subjects and objects connected via an unseen predicate with the help of only a few support set videos sharing the common predicate. These works underscore the critical role of temporal correlation in inferring dynamic visual relationships but overlook the essential prior knowledge of spatial-temporal correlations. Differently, we propose to incorporate spatial-temporal knowledge with multi-head cross-attention layers to learn more representative and discriminative feature representation to facilitate VidSGG.

Notably, exploiting an object detector to facilitate video understanding is common. Recently, Wang et al. [46] propose a novel SAOA framework to introduce the spatial location provided by the detector for egocentric action recognition, which aims to reason the interaction between humans and objects from an egocentric perspective. Although aligning local object features and location proposals to capture the spatial context, the SAOA framework ignores the crucial temporal correlation across continuous frames, which is the main distinction between it and our proposed STKET.

Recent advances in deep learning have allowed neural networks to learn potent representations from raw training data for various tasks [47, 38, 48, 49]. However, solely using these vanilla networks may achieve poor performance, especially in weakly supervised learning [50, 51] and domain adaption [52]. To address this challenge, lots of efforts have been made to integrate domain prior knowledge into deep representation learning, resulting in remarkable progress in numerous computer vision tasks, such as few-shot image recognition, facial expression recognition, visual navigation, scene graph generation [53, 54, 55, 56, 57, 36]. Specifically, Peng et al. [53] propose a novel knowledge transfer network that jointly incorporates visual feature learning, knowledge inferring, and classifier learning to fully explore prior knowledge in the few-shot task. Similarly, Chen et al. [54] propose a knowledge-guided graph routing framework, which unifies prior knowledge of statistical label correlations with deep neural networks for multi-label few-shot learning. Pu et al. [55] introduce the prior knowledge between action unit and facial expression to facilitate facial expression recognition. Yang et al. [57] incorporate the prior semantic knowledge into a deep reinforcement learning framework to address the semantic navigation task. Chen et al. [36] incorporate statistical correlations into deep neural networks to facilitate scene graph generation, in which these statistical correlations between object pairs and their relationships can effectively regularize semantic space and make prediction less ambiguous. However, most of these works merely consider prior spatial knowledge of statistic images, and VidSGG involves spatial-temporal contextual information. In this work, we propose to learn spatial-temporal knowledge and incorporate it into the multi-head cross-attention mechanism to learn more representative relationship representations to facilitate VidSGG.

When inferring visual relationships, humans leverage not only visual cues but also accumulated prior knowledge [60]. This approach has been validated on various vision tasks [61, 54, 56]. Inspired by this, we propose to distill the prior spatial-temporal knowledge directly from the training set for facilitating the VidSGG task. Specifically, we learn spatial co-occurrence and temporal transition correlations in a statistical manner, as shown in Figure 4.

To distill spatial co-occurrence correlations, the model learns the spatial knowledge embedding based on the spatial co-occurrence matrix. Specifically, the model generates the corresponding spatial knowledge embedding $\mathbf{s}^{k}_{t}$ for $k$-th relationship representation in the frame $I_{t}$, the generating process can be formulated as

where $s(k,t)$ and $o(k,t)$ denote the category of subject and object of $k$-th relationship representation in the frame $I_{t}$, $f_{spa}(\cdot)$ is implemented by four fully-connected layers which map a $C$-dimension vector to a 1936-dimension vector. For brevity, we denote the corresponding spatial knowledge embeddings for all object pairs in the frame $I_{t}$ by $\mathbf{S}_{t}=\{\mathbf{s}^{1}_{t},...,\mathbf{s}^{K(t)}_{t}\}$.

Since the distribution of real-world relationships is seriously unbalanced, directly utilizing these spatial knowledge embeddings may degrade the model performance on the less frequent relationships. Thus, we train these embeddings to predict predicates by using the binary cross-entropy loss as the objective function:

where $\ell(\cdot,\cdot)$ denote the binary cross-entropy loss function, $f(\cdot)$ denote the predicate classifier, $\mathbf{y}^{k}_{t}$ denote ground-truth predicate labels of $k$-th relationships in the frame $I_{t}$.

To explore temporal transition correlations, the model learns the corresponding temporal knowledge embedding based on the temporal transition matrix. Specifically, given the predicted object labels and relation labels, the model generates the corresponding temporal knowledge embedding $\mathbf{t}^{k}_{t}$ for $k$-th relationship representation in the frame $I_{t}$, the generating process can be formulated as

where $s(k,t)$ and $o(k,t)$ denote the category of subject and object of $k$-th relationship representation in the frame $I_{t}$, $r(k,t)$ denotes the predicate of $k$-th relationship representation in the frame $I_{t}$, $f_{tem}(\cdot)$ is implemented by four fully-connected layers which map a $C$-dimension vector to a 1936-dimension vector. Specifically, we utilize the spatial contextualized representation to coarsely predict the relation labels. For brevity, we denote the corresponding temporal knowledge embeddings for all object pairs in the frame $I_{t}$ by $\mathbf{T}_{t}=\{\mathbf{t}^{1}_{t},...,\mathbf{t}^{K(t)}_{t}\}$.

Since complicated predicate transition exists for different object category pairs, the temporal knowledge embedding may contain inaccurate temporal correlation, resulting in sub-optimal performance. Therefore, we train these embeddings to predict predicates at the next frame by using the binary cross-entropy loss as the objective function:

where we assume $k$-th relationship representation in the frame $I_{t}$ and frame $I_{t+1}$ belong the same subject-object pair for easily understanding.

Between object pairs and their relationships, there are apparent spatial co-occurrence correlations within each image and strong temporal transition correlations across different images. Thus, we propose incorporating spatial-temporal knowledge into the multi-head cross-attention mechanism to learn spatial- and temporal-embedded representations.

Spatial knowledge often encapsulates information about positions, distances, and relationships among entities. On the other hand, temporal knowledge concerns the sequence, duration, and intervals between actions. Given their unique properties, treating them separately allows specialized modeling to capture the inherent patterns more accurately. Therefore, we design spatial and temporal knowledge-embedded layers that thoroughly explore the interaction between visual representation and spatial-temporal knowledge.

Then, TKEL incorporates corresponding spatial and temporal knowledge embeddings in the keys and queries to fuse the information from the relationship representation and its corresponding spatial and temporal prior. Since we stack $N_{t}$ identical temporal knowledge-embedded layers, the input of the $n$-th layer is the output of $(n-1)$-th layer. Thus, the queries $\mathbf{Q}$, keys $\mathbf{K}$, values $\mathbf{V}$ and output of the $n$-th TKEL is presented as:

where $\mathbf{E}_{f}$ is the learned frame encoding vector, the $Att_{tem.}(\cdot)$ denote cross-attention layer in TKEL and the $\mathbf{W}_{Q}$, $\mathbf{W}_{K}$ and $\mathbf{W}_{V}$ denote the linear transformations. The intention of adding $[\mathbf{T}_{t-1},\mathbf{S}_{t}]$ into the queries and keys is to incorporate the temporal prior about the previous frame and the spatial prior about the current frame. In this way, the TKEL can effectively capture spatiotemporal context in the sliding window.

Considering that the relationships in a frame have various representations in different batches, we choose the earliest representation appearing in the sliding window. For simplicity, we remove the subscript $n$ and denote the final output of TKEL as $\mathbf{F}^{T}_{t}$.

As aforementioned, SKEL explores the spatial co-occurrence correlations within each image, and TKEL explores the temporal transition correlations across different images. Though fully exploring the interaction between visual representation and spatial-temporal knowledge, these two layers generate spatial- and temporal-embedded representations, respectively. To explore the long-term context information, we further design the spatial-temporal aggregation (STA) module to aggregate these representations for each object pair to predict the final semantic labels and their relationships. It takes the spatial- and temporal-embedded relationship representations of the identical subject-object pair in different frames as the input. Specifically, we concatenate these representations of the same object pair to generate context representation:

where $\mathbf{f}^{S}_{k,t}$ and $\mathbf{f}^{T}_{k,t}$ denote the spatial- and temporal-embedded representation for the $k$-th relationship in the frame $I_{t}$. And then, to find the same subject-object pair in different frames, we adopt the predicted object label and the IoU (i.e., intersection over union) to match the same subject-object pairs detected in frames $\{I_{t-\tau+1},...,I_{t}\}$ (more details in the supplementary material). Thus, the input of $k$-th relationship in the frame $I_{t}$ in the spatial-temporal aggregation module is presented as

where we assume $k$-th relationship representation in frame $\{I_{t-\tau+1},...,I_{t}\}$ belong same subject-object pair for easily understanding. And its corresponding output of the spatial-temporal aggregation module is presented as:

where $\mathbf{E}^{{}^{\prime}}_{f}$ is the learned frame encoding vector, the $Att(\cdot)$ denote the self-attention layer in the spatial-temporal aggregation module, and the $\mathbf{W}_{Q}$, $\mathbf{W}_{K}$ and $\mathbf{W}_{V}$ denote the linear transformations. Considering the relationships in a frame have various representations in different batches, we choose the earliest representation appearing in the sliding window.

In the real world, there exist different kinds of relationships between two objects at the same time. Thus, we introduce the binary cross-entropy loss as the objective function for predicate classification as follows:

For a subject-object pair $r$, $\mathcal{P}^{+}$ are the annotated predicates, while $\mathcal{P}^{-}$ is the set of the predicates not in the annotation. $\phi(r,p)$ indicates the computed confidence score of the $p$-th predicate.

During training, we adopt the binary cross-entropy loss as the objective function for supervising SKEL, TKEL, and STA and denote the corresponding loss by $\ell^{s}(r)$, $\ell^{t}(r)$ and $\ell^{c}(r)$ respectively. Therefore, the final classification loss is defined as summing the three losses over all samples, formulated as

Therefore, the total objective is formulated as:

In all experiments, we set the loss weights of the three losses to be equal, primarily to ensure that each loss component has an equal contribution to the overall optimization objective.

We use an AdamW [62] optimizer with initial learning rate $2e^{-5}$ and batch size 1 to train our model. Moreover, gradient clipping is applied with a maximal norm of 5. All experiments are implemented by PyTorch [63]. In the spatial-temporal aggregation module, we set the size of the sliding window $\tau$ to 4. In KEAL, we stack two identical spatial knowledge-embedded layers to explore spatial co-occurrence correlations and then stack two identical temporal knowledge-embedded layers to temporal transition correlations. The cross-attention and self-attention layers in our proposed framework have 8 heads with $d=1936$ and $dropout=0.1$. In SKEL and TKEL, the $1936$-dimension input is projected to $2048$-dimension by the feed-forward network, then projected to $1936$-dimension again after ReLU activation. In STA, the $3872$-dimension input is projected to $1936$-dimension.

To evaluate the effectiveness of STKET, we compare it with existing state-of-the-art VidSGG algorithms, including STTran [25], TPI [44], and APT [24]. Previous works [25] also adapt image-based SGG algorithms to address the VidSGG task by applying the inference process to each frame. We also follow these works to include the image-based SGG algorithms for more comprehensive evaluations and comparisons, including VCTREE [37], KERN [36], ReIDN [27], GPS-Net [28].

We first present the comparison on R@K in Table I. As shown, recent video-based algorithms (e.g., STTran, TPI, and APT) obtain quite a marginal improvement over the image-based SGG algorithms as they further introduce temporal contextual information. By introducing spatial-temporal prior knowledge to guide aggregating spatial and temporal contextual information, the proposed STKET framework consistently outperforms in nearly all settings. For example, it improves the R@10 from 78.5% to 82.6%, 55.1% to 57.1%, and 25.7% to 27.9% on the three tasks, with the improvement of 4.1%, 2.0% and 2.2%, respectively. It also obtains similar improvement on R@20 and R@50 metrics.

Considering the mR@K metric offers a better performance measure under uneven distribution [36], we also present the comparison results on this metric. As presented in Table II, STKET obtains even more significant improvement, enhancing mR@50 by 8.1%, 4.7%, and 2.1% compared with the current best-performing APT algorithm. To highlight the necessity of introducing the mR@K metric, we present the distribution across different relationships on the AG dataset in Figure 6a. As depicted, the distribution of relationships is exceedingly long-tailed, in which the top-10 most frequent relationships occupy 90.9% samples while the top-10 least frequency relationships merely occupy 2.1%. In Figure 6b, we further provide the detailed performance of each relationship to understand the performance variance across different relationships better. Evidently, current algorithms like STTran and APT deliver competitive performance for relationships with abundant training samples (e.g., “in front of”, “not looking at”) but falter significantly for relationships with limited training samples (e.g., “wiping”, “twisting”). In contrast, for the top-10 least frequent relationships, STKET improves the R@50 from 9.98% to 23.98% compared to the second-best APT and from 12.10% to 30.65% compared to the baseline STTran. These comparisons demonstrate STKET can effectively regularize VidSGG training and thus reduce the dependencies on training samples by explicitly incorporating spatial-temporal prior knowledge.

In Figure 5, we visualize the qualitative results of our method and current leading VidSGG methods (i.e., STTran and APT). The results show that existing state-of-the-art VidSGG algorithms tend to predict numerous false-positive relationships, while our method successfully predicts nearly all relationships with high accuracy. This highlights the strength of integrating spatial-temporal knowledge in discerning dynamic relationships, especially within intricate interactions. For instance, across all frames, our STKET accurately identifies the attentional relationship among person, cup, and laptop, where current leading VidSGG algorithms stumble. It is also noteworthy that our method not only performs well on high-frequency relationships but also on low-frequency relationships. For instance, in the last frame, our method correctly predicts the predicate of “drink from” whose samples merely occupy 0.355% of the total samples, while other methods miss it. This success is largely due to our method’s explicit incorporation of spatial-temporal correlations, which helps to lessen the dependency on training samples significantly.

In this section, we conduct comprehensive experiments to analyze the actual contribution of each crucial component. Here, we mainly present the R@10, R@20, mR@10, and mR@20 on Pred Cls and SG Cls as they can better describe the performance.

As aforementioned, our STKET framework leverages spatial-temporal prior knowledge to effectively regularize relationship predictions, thereby lessening reliance on training samples and mitigating the imbalance problem in the Action Genome. The significant performance boost within tailed relationship categories demonstrates it, as shown in Figure 6b. Nevertheless, in extreme scenarios where subject-predicate-object triplets are exceedingly rare, the STKET framework struggles to provide accurate statistical regularization and thus may predict false predicates. As illustrated in 9, the bias from skewed long-tail distribution leads to a common but incorrect predicate “sit on” instead of the rare but correct predicate “lean on”, in the relationship of “person-lean on-sofa”. The same type of misclassification also occur with ”person-carry-cloth”. For these challenges, we conjecture that integrating spatial-temporal knowledge from various distributions (e.g., ImageNet-VidVRD [21], Home Action Genome [64]) can provide more accurate and robust regularization. Furthermore, we argue that leveraging the abundant contextual information in web text data could enhance the regularization of such rare relationships.