Keyword-Aware Relative Spatio-Temporal Graph Networks for Video Question Answering

To correctly answer the question based on a given video, the agent is required to reason about the complex interactions between objects. As videos usually contain multiple objects, it is important to attend to question-relevant objects, which can be seen as keywords, without having the noise from irrelevant objects. However, most existing VideoQA methods with object-level relation modeling treat all the objects equally without considering their relevance to the keywords. Consequently, the redundant information from irrelevant objects may reduce reasoning accuracy. To overcome this limitation, we design a keyword attention module to identify and augment relevant object features for the subsequent relation reasoning. Specifically, we first compute the attended question feature $\hat{\bm{E}}\in\mathbb{R}^{1\times 300}$ with the attention mechanism,

where ${\bm{A}}_{w}\in\mathbb{R}^{L\times 1}$ is the attention weights. Then, we can use the attended question feature to guide the generation of object attention,

where $\sigma$ is the sigmoid function, ${\bm{W}}_{q}\in\mathbb{R}^{C_{o}\times 300}$ and ${\bm{A}}_{o}\in\mathbb{R}^{TK\times 1}$. Thus, the keyword-relevant objects are highlighted by assigning higher attention scores. Consequently, these objects will be attended in relation reasoning. Finally, the object node features are generated as follows,

where $\odot$ denotes the Hadamard product and ${\bm{V}}\in\mathbb{R}^{TK\times C_{o}}$.

Most existing VideoQA methods model the complex object relations by representing a video as a holistic spatio-temporal graph over all the detected object proposals [10, 11]. These methods usually simply aggregate neighboring nodes with affinity metrics, in which the relative relation information is largely ignored. However, relations are relative. Choosing different objects of interest may lead to different semantic and position relation reasoning. Taking the example of a writer writing a book, if the writer is chosen as the subject and the book as the object, the relation is “write”. Conversely, the relation is “author”, i.e., the book’s author is the writer. Similarly, for the position relation, a cup on the table can also be understood as the table under the cup. Therefore, we integrate the relative relation modeling into our graph.

Let ${\bm{v}}_{t}^{i}\in\mathbb{R}^{1\times C_{o}}$ denote the feature of the $i$-th object or node in the $t$-th frame of ${\bm{V}}$. Our relative-relation-augmented graph models the relations between the object and its neighbors as follows,

where ${\mathcal{N}}({\bm{v}}_{t}^{i})$ denotes the k-nearest neighbors of ${\bm{v}}_{t}^{i}$ in the representation space, ${\bm{W}}_{a}\in\mathbb{R}^{C\times C_{o}}$ is for absolute relation modeling and ${\bm{W}}_{r}\in\mathbb{R}^{C\times C_{o}}$ is for relative relation reasoning. Thus, our graph will be able to realize the relativity in relations.

Spatial and temporal relations are two different types of relations. Modeling them together may confuse networks and reduce the reasoning efficacy. Moreover, representing a video as a graph over all the objects is computationally expensive. Efficient information flow in such a large graph is challenging. Therefore, we propose to disentangle the graph in Eq. (7) into a spatial graph and a temporal graph,

where ${\bm{W}}_{a}^{s},{\bm{W}}_{r}^{s},{\bm{W}}_{a}^{t},{\bm{W}}_{r}^{t}\in\mathbb{R}^{C\times C_{o}}$ are the parameters for spatial and temporal reasoning.

In the spatial reasoning and aggregation parts, we use the $\max$ pooling operation to keep the most relevant nodes to the question and ignore the noise from the other irrelevant objects. In the temporal reasoning part, we employ the sum pooling operation. This is because in a short video clip, the object of interest usually appears in the entire video and the spatial relation in each frame is important to the overall dynamics reasoning. As shown in experiments, this max-sum pooling combination can achieve better accuracy than others, also demonstrating the benefits of this disentangled spatio-temporal modeling.

To generate the final output vector for answer prediction, we use bilinear attention [42] to project spatial graph presentations $(\{{\bm{s}}_{t}^{i}\},{\bm{Q}}_{w})$ and temporal graph presentations $(\{{\bm{t}}_{t}\},{\bm{Q}}_{w})$ to the same space as the question word-level features ${\bm{Q}}_{w}$’s $\mathbb{R}^{L\times C_{w}}$, respectively. Then, we employ the attention mechanism in [11] to fuse the projected spatial features, temporal features and the question sentence-level features ${\bm{Q}}_{s}$ to a vector, which is finally used to answer the question.

VideoQA generally includes two types of tasks: multi-choice and open-ended. In this section, we design the answer decoders to deal with different tasks. The multi-choice task aims to choose the correct answer from $M$ candidates. In this case, we concatenate the question with each answer and obtain $M$ question-answer sequences. For each sequence, we feed it into the network to compute the final output vector, and then employ a fully-connected layer to generate the predicted score $a_{i}$. Suppose $a^{+}$ is the score of the correct answer and $(a_{1}^{-},\dots,a_{M-1}^{-})$ are the scores of wrong answers. We use the pairwise hinge loss $\sum_{i=1}^{M-1}\max\big{(}0,1-(a^{+}-a_{i}^{-})\big{)}$ to train networks.

The open-ended task aims to choose the correct answer from a pre-defined answer set, which can be treated as a multi-label classification task. Therefore, we feed the output of our GNN into a classifier with two fully-connected layers to compute the class probabilities and train the network using the cross-entropy loss. For counting, i.e., open-ended numbers, we treat it as a regression task and train the network using the Mean Squared Error (MSE) loss.

We evaluate the proposed method on the three most commonly used benchmarks for VideoQA task.

We extract videos at $10$ frames per second for all the datasets, and uniformly sample $T$ frames from each video. According to the average video length, we set $T$ as $30$ for MSVD-QA & MSRVTT-QA datasets and $20$ for TGIF-QA dataset. For the global context feature extraction, we apply ResNet-152 model pretrained on the ImageNet [48] dataset for appearance features and I3D model pre-trained on the Kinetics [39] dataset for motion features. For the local feature extraction, we select $K=10$ object bounding boxes with the highest confidence scores for each frame, where the bounding boxes are generated using Faster R-CNN [41] pretrained on the Visual Genome [49] dataset. We set the number of graph layers $H$ in both spatial and temporal graphs as $2$ to capture spatial and temporal information. The ratio $\alpha$ to control number of neighboring nodes in spatial and temporal graphs is set as $0.6$ and $0.8$, respectively. Moreover, the model’s hidden size $d$ is set as $512$.

The proposed model is trained on PyTorch [50]. During training, we set batch size as $64$ for multi-choice questions and $128$ for open-ended and count questions. The dropout ratio is set as $0.3$ to prevent overfitting. We train the model for $30$ epochs using the Adam optimizer with a constant learning rate of $10^{-4}$. Experiments are implemented on Ubuntu 20.04 by NVIDIA GTX 3090 GPUs.

Table I and Table II present the performance comparison of our KRST with multiple SOTA methods on three popular benchmarks. The comparing methods can be classified into three categories: (1) attention-based methods, including ST-TP [43], Co-mem [29], PSAC [44], QueST [45] and HCRN [6]; (2) graph-based methods on frame relations, including HGA [7] and B2A [8]; (3) graph-based methods on object relations, including L-GCN [9], GMN [10], HOSTR [36], MASN [11] and HQGA [46]. The experimental results demonstrate that our model consistently outperforms the SOTA models on three datasets. Note that it is challenging to achieve the best accuracies on all of the three datasets. For example, MASN [11] achieves quite a high accuracy on the TGIF-QA dataset but only reaches inferior accuracies on the MSVD-QA and MSRVTT-QA datasets. Similarly, HQGA [46] can achieve satisfactory performance on MSVD-QA and MSRVTT-QA but not on TGIF-QA (especially for the action task). In contrast, experiments show the superiority of our method on the three datasets, which implies that our method can effectively capture complex object interactions in space and time for VideoQA.

HGA and B2A are graph-based methods that model the temporal relations between different frames. In these methods, multiple graphs are constructed to exploit the correlation between question words and video segments. However, they ignore the fine-grained object-level information important for question answering, leading to inferior performance.

GMN and HOSTR build a holistic spatio-temporal graph to model object relations by taking object trajectories as input. They assume objects exist throughout the video, which may be invalid for videos with multiple objects and heavy occlusions. This assumption may bring extra noise in object relation modeling and result in inferior performance. MASN and HQGA are similar to our work in taking independent object proposals as input. However, both MASN and HQGA perform relation reasoning ignoring the keywords from questions, and they do not consider the relative relation between objects. By filling these gaps, our model demonstrates a clear superiority over these SOTA methods on all the three datasets.

In this section, we conduct a set of ablation studies on TGIF-QA dataset. First, we study the impact of important hyper-parameters (number of neighbors in graphs and number of graph layers). Then, we compare different pooling methods in graph construction. Lastly, we verify the effectiveness of each proposed component.

We visualize the learned attention maps (word attention ${\bm{A}}_{w}$ and object attention ${\bm{A}}_{o}$) and the k-nearest neighbors of the object. In Fig. 5 (a), we can observe that the keywords such as “dog”, “cat” and “food” are highlighted by higher attention weights. These highlighted keywords can help to identify the question-relevant objects for the subsequent relation reasoning. In Fig. 5 (b), we present the detected objects with their learned attention weights (red numbers) which are also indicated by the brightness of the object regions. It is shown that the keyword-relevant objects (e.g., “dog”, “cat” and “food”) have higher attention weights and therefore are much more brighter than the keyword-irrelevant objects (e.g., “closet”, “oven” and “jar”). This implies that the most keyword-relevant objects are effectively identified from the video frames by object attention. In Fig. 5 (c), we visualize the k-nearest neighbors of the object “dog”. It is observed that our model can correctly select the most relevant objects as neighbors, which help improve the efficacy of relation reasoning.

We also compare our method with the existing SOTA methods on different tasks of the TGIF-QA dataset in Fig. 6. The comparing methods include two graph-based methods: 1) HGA [7] on frame relations and 2) MASN [11] on object relations. It is observed that our model can correctly answer the questions based on given videos, which demonstrates the efficacy of our model in object relation reasoning. In Fig. 6 (a), our model can attend to the relevant object (i.e.,“blue ball”) from other distracting objects (e.g., “green ball”) and generate the correct answer. This validates the effectiveness of the keyword attention module in filtering out the question-irrelevant features from videos. In Fig. 6 (b), the example shows that our model can correctly predict the answer to the question, which requires the modeling of both spatial relations (e.g., “pick up bottle”) and temporal dynamics (e.g., “after”). In Fig. 6 (c), our model correctly answers the question by aggregating the information in multiple frames. The reason could be that we apply max pooling to aggregate information from neighboring frames in the temporal graph to maintain critical spatial relations in each frame. In Fig. 6 (d), our model generates the correct answer by attending to the question-relevant object (i.e., “man with a black shirt”). This again demonstrates the effectiveness of the proposed keyword attention module. In Fig. 6 (e), all the three models generate incorrect predictions for this sample. The primary cause could be the object detector’s inability to identify all the individuals presenting in the video, due to the heavy occlusions and poor lighting conditions.