Fully Convolutional Scene Graph Generation

Philosophers, linguists and artists have long wondered about the semantic content of what the mind perceives in images and speech [2, 9, 31, 45]. Many have argued that images carry layers of meaning [3, 47]. Considered as an engineering problem, semantic content has been modeled either as latent representations [12, 18, 27, 40], or explicitly as structured representations [39, 57, 60]. For a computer vision system to explicitly represent and reason about the detailed semantics, Johnson [26] et al.adopt and formalize scene graphs from computer graphics community. A scene graph serves as a powerful representation that enables many down-stream high-level reasoning tasks such as image captioning [63, 66], image retrieval [25, 26], Visual Question answering [22, 52] and image generation [25, 61].

A scene graph is considered as an explicit structural representation for describing the semantics of a visual scene. The nodes in a scene graph represent the object classes and the edges represent the relationships between the objects. Figure 1(a) shows a simple example of a scene graph that represents the underlying semantics of an image. Each relationship between two objects is denoted as a triplet of <subject, PREDICATE, object> , i.e., banana $\xrightarrow{\texttt{ON}}$ chair and chair $\xrightarrow{\texttt{HAS}}$ wheel in Figure 1(a). Most of the SGG work [6, 50, 51, 60, 70] is build as a two-stage pipeline: object detection then scene graph generation. For the first stage of object detection, existing object detectors, \ie, Fast/Faster R-CNN [13, 43], are used for object feature extraction from region proposals. For the second stage of scene graph generation, various approximation methods [51, 60, 70] for graph inference have been used. Some work [6, 67, 68, 69] have also investigated how to utilize external knowledge for improving the results. However, most previous work suffer from not only the long-tailed distribution of relationships [7, 11, 70], but also the highly biased prediction conditioned on object labels [16, 23, 28, 50]. Consequently, frequent predicates will prevail over less frequent ones, and unseen relationships can not be identified. Moreover, the extensibility and inference speed of a SGG framework is crucial for accelerating down-stream tasks. Although few researchers have studied the efficiency and scalability in SGG [14, 33, 62], the high computational complexity impedes the practicality towards real-world applications. A natural question that arises is: can we solve scene graph generation in a per-pixel prediction fashion? Recently, anchor-free object detectors [30, 53, 65, 72] have become popular due to their simplicity and low cost. These methods treat an object as a single or many, pre-defined or self-learned keypoints. Relating object detection to human pose estimation, if an object can be modeled as a point (human “keypoint”), is it possible to represent a binary relationship as vectors (human “limb”)?

In this paper, we propose a novel fully convolutional scene graph generation model, \ie, FCSGG, with state-of-the-art object detection results on Visual Genome dataset [29], as well as compelling SGG results compared with visual-only methods. We present a bottom-up representation of objects and relationships by modeling objects as points and relationships as vectors. Each relationship is encoded as a segment in a 2D vector field called relation affinity field (RAF). Figure 1(c) shows an illustration of RAFs for predicates ON and HAS. Both objects and relationships are predicted as dense feature maps without losing spatial information. For the first time, scene graphs can be generated from a single convolutional neural network (CNN) with significantly reduced model size and inference speed. Specifically, we make the following contributions:

• •

  We propose the first fully convolutional scene graph generation model that is more compact and computationally efficient compared to previous SGG models.

• •

  We introduce a novel relationship representation called relation affinity fields that generalizes well on unseen visual relationships. FCSGG achieves strong results on zero-shot recall.

• •

  Our proposed model outperforms most of the visual-only SGG methods, and achieves competitive results compared to methods boosted by external knowledge.

• •

  We conduct comprehensive experiments and benchmark our proposed method together with several previous work on model efficiency, and FCSGG achieves near real-time inference.

We categorize the related work of SGG into the following directions: refinement of contextual feature, adaptation of external knowledge, and others.

Contextual feature refinement. Xu et al. [60] proposed an iterative message passing mechanism based on Gated Recurrent Units [8], where the hidden states are used for predictions. Followers [51, 70] studied better recurrent neural networks [21, 44, 48] for encoding object and edge context. Others trying to incorporate more spatial features into SGG. Li et al. [34] proposed the MSDN that merges features across multiple semantic levels, and later achieved message passing constrained on visual phrase [32]. Dai et al.[10] proposed a spatial module by learning from bounding box masks. Woo et al. [56] introduced the geometric embeddings by directly encoding the bounding box offsets between objects. Wang et al.[55] further studied the effects of relative positions between objects for extracting more discriminating features. Our method is fundamentally different from these methods as the relationships are grounded semantically and spatially directly into CNN features. Without any explicit iterative information exchange between nodes and edges, our model is able to predict objects and relationships in a single forward pass.

External knowledge adaptation. Beyond visual features, linguistic knowledge can serve as additional features for SGG [15, 39, 42, 67]. By adopting statistical correlations of objects, Chen et al. [6] utilized graph neural networks [46] to infer relationships. Gu et al. [17] and Zareian et al.[68, 69] explored the usefulness of knowledge or commonsense graphs for SGG. Tang et al. [50] proposed an de-biasing method by causal interventions of predictions. Lin et al. [38] investigated the graph properties and mitigated the long-tailed distributions of relationships. Compared with these methods, our proposed model relies only on visual features but still yields a strong performance.

Very few researchers have investigated alternatives either for object feature or relationship feature representations. Newell [41] and Zhang et al. [71] tried latent-space embeddings for relationship and achieved improvements. Different from most of the previous work, FCSGG reformulates and generalizes relationship representations from only visual-based features in near real-time, which is much faster than specifically designed SGG models for efficiency [33, 62].

In this section, we provide the preliminaries of modeling object detection as keypoint estimation in a single-scale dense feature map prediction fashion.

Our model is built upon a one-stage anchor-free detector, namely CenterNet [72]. Different from commonly used anchor-based R-CNN approaches for generating object proposals and features, it predicts three dense features that represent centers of object bounding boxes, center offsets, and object sizes. More specifically, an input image $\mathrm{I}\in\mathbb{R}^{3\times W\times H}$ will go through a backbone CNN generating feature maps of size $w\times h=\lfloor\frac{W}{\tau}\rfloor\times\lfloor\frac{H}{\tau}\rfloor$ where $\tau$ is the total stride until the last layer, and we set $\tau=4$ unless specified otherwise. Then these features will be fed into three prediction heads, each of which consists of several convolutional layers. The three heads are for predicting object center heatmaps $\mathbf{O}\in\mathbb{R}^{\mathcal{C}\times w\times h}$ where $\mathcal{C}$ is the number of object classes in a dataset, object center offsets $\mathbf{\Delta}\in\mathbb{R}^{2\times w\times h}$ for recovering from downsampled coordinates, and object sizes $\mathbf{S}\in\mathbb{R}^{2\times w\times h}$, respectively (shown in the dashed block of Figure 2). We define the ground-truth (GT) objects in an image as $\mathbf{B}=\{b^{i}\}$ where $b^{i}=(x_{0}^{i},y_{0}^{i},x_{1}^{i},y_{1}^{i},c^{i})$ is the object $i$ of class $c^{i}$, $(x_{0}^{i},y_{0}^{i})$ and $(x_{1}^{i},y_{1}^{i})$ denote the coordinates of the left-top and right-bottom corners of its bounding box. The center of the bounding box is defined as $\mathbf{o}^{i}=(\mathbf{o}_{x}^{i},\mathbf{o}_{y}^{i})=((x_{0}^{i}+x_{1}^{i})/2,(y_{0}^{i}+y_{1}^{i})/2)$, and the size of the object is defined as $\mathbf{s}^{i}=(x_{1}^{i}-x_{0}^{i},y_{1}^{i}-y_{0}^{i})$. To obtain the ground-truth center heatmaps at feature level, we divide coordinates by the stride $\tau$ and add Gaussian-smoothed samples following Law [30] and Zhou et al. [72]. Formally, the object center $\mathbf{o}^{i}$ will be modulated by a bivariate Gaussian distribution along x-axis and y-axis on $\mathbf{O}_{c^{i}}$. The value around $\mathbf{o}^{i}$ is computed as

where $\sigma_{x}$ and $\sigma_{y}$ controls the spread of the distribution. When multiple objects of the same class $c$ contribute to $\mathbf{O}_{c,x,y}$, the maximum is taken as the ground truth. The center heatmaps are then supervised by Gaussian focal loss [30, 36, 72]. More details are provided in the supplementary file.

In addition to the supervision of center heatmaps, the center offset regression $\mathcal{L}_{\mathbf{\Delta}}$ and object size regression $\mathcal{L}_{\mathbf{S}}$ are used to recover object detections. For mitigating discretization error due to downsampling, the offset target is $\boldsymbol{\delta}^{i}=\mathbf{o}^{i}/\tau-\lfloor\mathbf{o}^{i}/\tau\rfloor$, and regressed via L1 loss as $\mathcal{L}_{\mathbf{\Delta}_{x,y}}\text{=\;}||\hat{\mathbf{\Delta}}_{x,y}-\boldsymbol{\delta}^{i}||_{1}$ at center locations. For object size regression $\mathcal{L}_{\mathbf{S}}$, the target is feature-level object size $\mathbf{s}^{i}/\tau$, and the actual size can be recovered by multiplying the output stride. We also use L1 loss as $\mathcal{L}_{\mathbf{S}_{x,y}}\text{=\;}||\mathbf{\hat{S}}_{x,y}-\mathbf{s}^{i}||_{1}$ at center locations. Both object size and offset regressors are class-agnostic such that there will be only one valid regression target at a particular location where $\mathbf{O}_{c,x,y}=1$. If two object centers collide onto the same location, we choose the smaller object for regression. The overall object detection objective is

where $N$ is the total number of objects in the image, $\lambda_{\mathbf{\Delta}}$ and $\lambda_{\mathbf{s}}$ are hyper-parameters for weight balancing. We empirically set $\lambda_{\mathbf{\Delta}}=1$ and $\lambda_{\mathbf{s}}=0.1$ for all experiments. Until here, object centers, offsets, and sizes are all represented in single-scale feature maps. We will discuss a multi-scale prediction approach reducing regression ambiguity effectively in section 4.2.

Newell and Deng [41] model objects as center points, and ground edges at the midpoint of two vertices then construct the graph via associative embeddings. The midpoint serves as a confidence measurement of presence of relationships. However, false detections and ambiguities arise when there are crowded objects in a region, or the associated objects of a relation are far away from each other. Another limitation is that it still needs feature extraction and grouping that cause low inference speed. Inspired and from a bottom-up 2D human pose estimation work called OpenPose [4], we migrate the concept of part affinity fields into scene graph generation. Our proposed method grounds relationships onto CNN features pixel by pixel, and mitigates above mentioned limitations.

Our model is conceptually simple: in addition to the outputs that are produced by the object detection network described in Section 3, we add another branch that outputs a novel feature representation for relationships called relation affinity fields (RAFs). Specifically, the RAFs are a set of 2D vector fields $\mathbf{F}=\{\mathbf{F}_{p}\}\in\mathbb{R}^{\mathcal{P}\times 2\times h\times w}$ , where $p\in\mathbb{R}^{\mathcal{P}}$ and $\mathcal{P}$ is the number of predicate classes in a dataset. Each 2D vector field $\mathbf{F}_{p}$ represents the relationships among all the object pairs of predicate $p$. Given our definition of objects as center points, the ground-truth RAFs are defined as vectors flow from the center of subject to the center of object. More formally, we define the binary relationships among objects $\mathbf{B}$ in the input image as $\mathbf{R}=\{r^{i\to j}\}$, where $r^{i\to j}=(b^{i},p^{i\to j},b^{j})$ is the relationship triplet from subject $b^{i}$ to object $b^{j}$ with predicate $p^{i\to j}$. We define a “path” $\pi_{p}^{i\to j}$ that “propagates” $p^{i\to j}$ from subject center $\mathbf{o}^{i}$ to object center $\mathbf{o}^{j}$. For a point $\mathbf{p}=(x,y)$, its ground-truth relation affinity field vector $\mathbf{F}_{p,x,y}$ is given as

and the path $\pi_{p}^{i\to j}$ is defined on a set of points between object centers forming a rectangular region:

where $\epsilon_{\mathbf{e}^{i\to j}}=||\mathbf{o}^{j}-\mathbf{o}^{i}||_{2}$ as the relationship “length” along the direction $\mathbf{e}^{i\to j}$, and $\epsilon_{\mathbf{e}_{\perp}^{i\to j}}=\mathrm{min}(a^{i},b^{i},a^{j},b^{j})$ as the relationship “semi-width” along $\mathbf{e}_{\perp}^{i\to j}$ (orthogonal to $\mathbf{e}^{i\to j}$) being the minimum of object centers’ radii. Since vectors may overlap at the same point, the ground-truth RAF $\mathbf{F}_{p}$ averages the fields computed for all the relationship triplets containing that particular predicate $p$. It is given as $\mathbf{F}_{p}=\frac{1}{n_{c}(x,y)}\sum_{x,y}\mathbf{F}_{p,x,y}$, where $n_{c}(x,y)$ is the number of non-zero vectors at point $(x,y)$. With the definition of ground-truth RAFs, we can train our network to regress such dense feature maps. The RAF regression loss $\mathcal{L}_{raf}$ can be estimated using a normal regression losses $\mathcal{L}_{reg}$ such as L1, L2 or smooth L1 [13]. Given the predicted RAFs $\hat{\mathbf{F}}$, the loss is defined as per-pixel weighted regression loss as

where $\mathbf{W}$ is a pixel-wise weight tensor of the same shape of $\mathbf{F}$. The weights $\mathbf{W}$ are determined and divided into three cases (Figure 3): a) $\mathbf{W}_{p,x,y}=1$if $(x,y)$ is exactly on the line segment between objects having the relationship $p$ b) $\mathbf{W}_{p,x,y}\in(0,1)$if the distance between $(x,y)$ and the line segment is small and the value is negative correlated to the distance c) otherwise where $\mathbf{F}_{p,x,y}=\mathbf{0}$. We provide ablation study on the choice of losses and the weight tensor in Section 5.3. Finally, the complete loss for training our proposed model can be written as $\mathcal{L}=\mathcal{L}_{det}+\mathcal{L}_{raf}$.

Our proposed RAFs encode rich information of both spatial and semantic features as dense feature maps, and enable end-to-end joint training of object detection and relation detection. To extract relationship from predictions, path integral over RAFs is performed which will be described below.

Dataset. We use the Visual Genome (VG) [29] dataset to train and evaluate our models. We followed the widely-used preprocessed subset of VG-150 [60] which contains the most frequent 150 object categories ($\mathcal{C}$ = 150) and 50 predicate categories ($\mathcal{P}$ = 50). The dataset contains approximately 108k images, with 70% for training and 30% for testing. Different from previous works [5, 51, 70], we do not filter non-overlapping triplets for evaluation.

General settings. We experiment on two settings, one for small input size ($\text{L}_{max}$ = 512) and one for larger size ($\text{L}_{max}$ = 1024). The model is trained end-to-end using SGD optimizer with the batch size of 16 for 120k iterations. The initial learning rate is set to 0.02 and decayed by the factor of 10 at 80k${}^{\text{th}}$ and 100k${}^{\text{th}}$ iteration. We adopt standard image augmentations of horizontal flip and random crop with multi-scale training. During testing, we keep the top 100 detected objects for path integral.

Metrics. We conduct comprehensive analysis following three standard evaluation tasks: Predicate Classification (PredCls), Scene Graph Classification (SGCls), and Scene Graph Detection (SGDet). We report results of recall@K (R@K) [39], no-graph constraint recall@K (ngR@K) [41, 70], mean recall@K (mR@K) [6, 51], no-graph constraint mean recall@K (ng-mR@K), zero-shot recall@K (zsR@K) [39] and no-graph constraint zero-shot recall@K (ng-zsR@K) [50] for all three evaluation tasks. We do not train separate models for different tasks.

Scene graph generation is a critical pillar for building machines to visually understand scenes and perform high-level vision and language tasks. In this paper, we introduce a fully convolutional scene graph generation framework that is simple yet effective with fast inference speed. The proposed relation affinity fields serve as a novel representation for visual relationship and produce strong generalizability for unseen relationships. By only using visual features, our exploratory method achieves competitive results over object detection and SGG metrics on the VG dataset. We expect that FCSGG can serve as a general and strong baseline for SGG task, as well as a vital building block extending to down-stream tasks.

This work was partially supported by Bourns Endowment funds at the University of California, Riverside.