Importance Weighted Structure Learning for Scene Graph Generation

As a structured prediction task, scene graph generation aims to build a visually-grounded scene graph for an input image by explicitly identifying the scene objects and their relevant relationships. In the current SGG settings, the scene graph includes a list of intertwined semantic triplet structures, in which each triplet consists of three components: a subject, a predicate and an object. Specifically, the current SGG tasks only focus on inferring the pairwise relationships, where the relationship between two interacting instances (subject and object) in an input image is termed as a predicate.

Generally, SGG task aims to infer the optimum interpretations $z^{*}$ from the input image $x$ via a MAP estimation $z^{*}=\operatorname*{arg\,max}_{z}p_{\theta}(z|x)$, in which the underlying posterior $p_{\theta}(z|x)$ can be computed as follows:

where $s_{\theta}(x,z)$ is a scoring function of the probabilistic graphical model (e.g. conditional random field [41]) of the SGG task, which is used to measure the similarity/compatibility between the input variable $x$ and the output variable $z$. $s_{\theta}(x)$ is the relevant partition function or normalizing constant. Due to the exponential combinatorial dependencies existing in the SGG task, $s_{\theta}(x)$ is generally computationally intractable, which implies that it can only be estimated by certain approximation strategies.

To this end, a classical variational inference (VI) [7], [8] technique is generally employed to estimate the computationally intractable log-partition function $logs_{\theta}(x)$ in current SGG models. For tractability, the computationally tractable variational distribution is often assumed to be fully decomposed and the resulting inference technique is known as mean field variational inference (MFVI) [7], [8]. Besides the above variational inference step, one still requires another variational learning step to fit the underlying posterior with the ground-truth training samples. Such formulation is also known as mean field variational Bayesian (MFVB) [7], [8] framework, in which the optimum $q^{*}(z)$ and $\theta^{*}$ are obtained by alternating the above inference and learning steps. More importantly, for MFVI, the original MAP inference in SGG task can be transformed into a corresponding marginal inference task, which is not the case for other VI techniques [42].

In the current SGG modelling approaches, a classical cross-entropy loss is generally employed to solve the variaitional learning step, while the above MFVI step is commonly formulated using a message passing neural network (MPNN) [43], [44], [45], [46] model, in which the traditional ELBO is invariably implicitly chosen as the variational inference objective. As a result, the MPNN-based MFVB formulation has became the de facto solution for current SGG tasks. In this MPNN-based MFVB framework, two fundamental modules are required, namely, visual perception and visual context reasoning. The former aims to locate and instantiate the objects and predicates within the input image, while the latter tries to infer their consistent interpretation.

Specifically, given an input image $x$, visual perception module aims to generate a set of object region proposals $b_{i}^{o}\in\mathbb{R}^{4},i=1,...,m$, as well as a set of predicate region proposals $b_{j}^{p}\in\mathbb{R}^{4},j=1,...,n$, where $m$ and $n$ represent the number of objects and predicates detected in the input image, respectively. Correspondingly, the input image $x$ can be divided into two sets of image patches $x_{i}^{o},i=1,...,m$ and $x_{j}^{p},j=1,...,n$. One can extract the relevant fixed-sized latent feature representation sets $y_{i}^{o}\in\mathbb{R}^{d},i=1,...,m$ and $y_{j}^{p}\in\mathbb{R}^{d},j=1,...,n$ by applying a ROI pooling on the feature maps obtained from the visual perception module. Given a set of object classes $\mathcal{C}$ and a set of relationship categories $\mathcal{R}$, a visual context reasoning module aims to infer the resulting object and predicate interpretation sets $z_{i}^{o}\in\mathcal{C},i=1,...,m$ and $z_{j}^{p}\in\mathcal{R},j=1,...,n$ based on the above latent feature representation sets.

However, the variational distribution derived from the above loose ELBO objective often underestimates the underlying posterior [21], which leads to inferior generation performance. To this end, in this paper, we propose a novel importance weighted structure learning (IWSL) method, which employs a tighter (which is explicitly explained in Section 3.4) importance weighted lower bound to replace the classical ELBO as the variational inference objective. Instead of relying on the classical message passing technique, the proposed IWSL method applies a generic entropic mirror descent algorithm to accomplish the resulting constrained variational inference task. The proposed generic IWSL methodology allows us to increase the model complexity so that one can find a balanced bias-variance trade-off. In other words, the resulting variational distribution derived from the tighter importance weighted lower bound objective facilitates a better exploration of the complex multiple-modal behaviour of the underlying posterior.

As a structured prediction task, scene graph generation can be generally formulated using a probabilistic graphical model, e.g. a conditional random field (CRF) [41]. With such probabilistic graphical model, one can model the conditional dependencies among the relevant variables by devising a non-negative scoring function $s_{\theta}(x,z)$, where $\theta$ is used to parameterize the scoring function.

Basically, $s_{\theta}(x,z)$ measures the similarity or compatibility between the input image $x$ and the output interpretations $z$, which is often defined as follows:

where $r$ is a clique within a clique set $R$ (which is defined by the corresponding graph structure), $f_{r}$ is a non-negative factor function which models the dependencies between $x_{r}$ and $z_{r}$. Generally, the underlying posterior $p_{\theta}(z|x)$ related to $s_{\theta}(x,z)$ is often assumed to be a Gibbs distribution in traditional VB framework [7]. Correspondingly, $f_{r}$ often takes the exponential form and the log scoring function is computed as follows:

where $\psi_{r}$ is also known as a potential function. In current SGG models, we have two types of potential functions: the unary potential function $\psi_{u}$ and the pairwise/binary potential function $\psi_{b}$.

Currently, only local contextual information is considered in most previous SGG models, while the global contextual information is largely ignored. In this paper, we aim to compute a latent global feature representation $y^{g}\in\mathbb{R}^{d}$ from the global region proposal $b^{g}$, where $b^{g}$ is obtained by the union of all the associated object/predicate region proposals in the input image. Correspondingly, $x^{g}$ is the relevant global image patch of $b^{g}$, and $z^{g}$ is its interpretation.

With the above definitions, by adding two types of pairwise potential terms $\psi_{b}^{p}(x_{j}^{p},x^{g},z_{j}^{p},z^{g})$ and $\psi_{b}^{o}(x_{i}^{o},x^{g},z_{i}^{o},z^{g})$, one could incorporate the global contextual information into the following applied log scoring function:

where the superscripts $o$, $p$, $g$ represent the object, the predicate and the global context, respectively. $N(i)$ is the set of neighbouring nodes around the target $i$. It is worth to note the latent feature representations $y$ are implicitly embedded in the above formulation.

In SGG tasks, the output variables $z$ are generally assumed as categorical variables, which are rarely applied in stochastic neural networks due to the inability to compute and backpropagate the gradients of the associated discrete distributions [23], [24]. To this end, instead of generating non-differentiable samples from a categorical distribution, a Gumbel-Softamx sampler [23], [24] is employed to produce differentiable samples drawn from a novel Gumbel-Softmax distribution. Such sampler is generally reparameterizable, in which an efficient gradient estimator can be easily implemented.

Suppose $z$ be the interpretation of a potential region proposal, it can be modelled as a categorical variable with the class probabilities $\pi^{1},...,\pi^{v}$ (where $v$ is the number of hypotheses for $z$), and the output categorical variables in SGG are encoded as $v$-dimensional one-hot vectors locating on the corners of the $(v-1)$-dimensional simplex, $\Delta^{v-1}$. The reparameterization function $g_{\pi}$ in Gumbel-Softmax sampler is defined as follows:

where $\sigma$ represents a $v$-dimensional Gumbel noise, $z\in\Delta^{v-1}$ is the output $v$-dimensional sample vector and its $i$-th element $z^{i}$ is computed as follows:

where $\tau$ is the softmax temperature. The samples drawn from the Gumbel-Softmax sampler become one-hot vectors when annealing the softmax temperature $\tau$ to zero. In reality, $\tau$ is often annealed to a relatively low temperature instead of zero.

In traditional VB framework, ELBO is generally employed as the variational inference objective and the resulting maximization of ELBO is used to approximate the computationally intractable log-partition function [21]. Given a scoring function $s_{\theta}(x,z)$ and a computationally tractable variational distribution $q(z)$, one can easily derive the following:

where, on the right-hand side of the equation, the first term is the so-called ELBO and the second term is the Kullback–Leibler (KL) divergence between the variational distribution $q(z)$ and the underlying posterior $p_{\theta}(z|x)$. Clearly, ELBO is lower bound of $logs_{\theta}(x)$ since the KL divergence term is non-negative. However, such loose ELBO objective often leads to overly simplified model, which may not capture the complicated multi-modal structure of the underlying posterior [22].

To this end, a tighter lower bound $\mathcal{L}_{s}$ based on $s$-sample importance weighting [22] is employed to replace the classical ELBO in this paper. Such lower bound is also known as importance weighted lower bound, which is defined as follows:

where $s$ represents the number of samples, in which each $z_{i}$ is an $i.i.d.$ random sample drawn from the variational distribution $q(z)$. $w_{i}=\frac{s_{\theta}(x,z_{i})}{q(z_{i})}$ is also known as the importance weight, and $\mathcal{L}_{s}$ is the importance weighted lower bound of $logs_{\theta}(x)$ when $s$ is a relatively small value. Essentially, $\mathcal{L}_{s}$ becomes an unbiased estimator of $logs_{\theta}(x)$ when $s$ reaches infinity. In particular, the traditional ELBO is just a special case of the importance weighted lower bound $\mathcal{L}_{s}$ (when setting $s=1$). Using more samples could only improve the tightness of the bound [22]. Therefore, compared with the traditional ELBO, $\mathcal{L}_{s}$ is a much tighter lower bound of the log partition function $logs_{\theta}(x)$.

For tractability, the above variational distribution $q(z)$ is often assumed to be fully decomposed as:

where $q^{o}_{i}(z_{i}^{o})\in\Delta^{v_{o}-1}$ and $q^{p}_{j}(z_{j}^{p})\in\Delta^{v_{p}-1}$ are local variational approximations for the objects and predicates in the output scene graph, respectively. $v_{o}$ and $v_{p}$ are the sizes of vocabularies for the objects and predicates, respectively. Such inference procedure is also known as mean field variational inference (MFVI) [7], [8]. In MFVI, the MAP inference formulated in the first subsection can be transformed into a corresponding marginal inference task, which may not be the case in general [42].

Given a potential region proposal $b_{i}$, the corresponding local log marginal posterior $logp_{\theta}(z_{i}|x_{i})$ is computed as follows:

where $logs_{\theta}(x_{i})$ is the computationally intractable log partition function, and $logs_{\theta}(x_{i},z_{i})=\sum_{z\backslash z_{i}}logs_{\theta}(x_{i},z)$ (where $z\backslash z_{i}$ represents marginalization over all nodes except the node $i$) is the local log marginal scoring function of $b_{i}$, which is generally obtained by a variable elimination technique.

Specifically, for a potential object region proposal $b_{i}^{o}$, it is computed as follows:

while for a potential predicate region proposal $b_{j}^{p}$, it is obtained by:

In the above two equations, $\cdot$ means an inner product, $z_{i}^{o}$ and $z_{j}^{p}$ are the output variables for an object and a predicate, which are generated by a Gumbel-Softmax sampler. $\mathcal{G}$ is a relevant global region proposal interpretation set. The feature representation learning functions $h^{o}_{\theta}$, $h^{p}_{\theta}$, $g^{op}_{\theta}$, $g^{oo}_{\theta}$, $g^{og}_{\theta}$, $g^{po}_{\theta}$, $g^{pg}_{\theta}$ are constructed by combing visual perception modules and multi-layer perceptrons (MLPs), which are parameterized by $\theta$. Each of these functions will first map the input image patches $x$ into the corresponding feature representations $y\in\mathbb{R}^{d}$ via the visual perception module, and then obtain the resulting $\mathbb{R}^{v}$ dimensional feature vector by feeding relevant $y$ into the corresponding MLP. Most importantly, the MLPs implicitly perform the potential function marginalization prescribed in Equations (11) and (12). The resulting log score is essentially the inner product of the above $\mathbb{R}^{v}$ dimensional feature vector and the corresponding $v$-dimensional vector $z$.

To approximate the computationally intractable $logs_{\theta}(x_{i})$, in this paper, we employ the following constrained variational inference objective $\mathcal{L}_{s}^{i}$:

where $\mathcal{L}_{s}^{i}$ represents the $s$-sample importance weighted lower bound, and the local variational approximation $q_{i}(z_{i})$ is set to a Gumbel-Softmax distribution with a categorical probability $\pi_{i}\in\Delta^{v-1}$. $z_{i1},...,z_{is}$ represent the $s$ $i.i.d$ samples drawn from $q_{\pi_{i}}(z_{i})$. Since the local Gumbel-Softmax variational distribution $q_{\pi_{i}}(z_{i})$ is reparameterizable, based on the Gumbel-Softmax sampler in the above subsection, $\mathcal{L}_{s}^{i}$ can be formulated as follows:

where $\sigma_{ij}$ is $v$-dimensional Gumbel noise drawn from a Gumbel distribution $u(\sigma_{i})$, which is fed into the Gumbel-Softmax reparameterization function $g_{\pi_{i}}$ to explicitly compute the corresponding output sample $z_{ij}$.

As a result, the above expectation $\mathcal{L}_{s}^{i}$ can be approximated using a Monte Carlo estimator as follows:

where the efficient computation of the log importance weight $logw_{ij}=logs_{\theta}(x_{i},z_{ij})-logq_{\pi_{i}}(z_{ij})$ is essential for computing the above $\mathcal{L}_{s}^{i}$. Specifically, $logs_{\theta}(x_{i},z_{ij})$ can be easily computed based on Equation (11) and (12), while the above log probability $logq_{\pi_{i}}(z_{ij})$ is approximated as follows:

where $\lVert.\rVert_{1}$ represents the $\mathbb{L}_{1}$ norm while $max(\pi_{i})$ is the maximum value of $\pi_{i}$.

With the above $\mathcal{L}_{s}^{i}$, one can compute the target log marginal posterior $logp_{\theta}(z_{i}|x_{i})$ via a corresponding surrogate logit $\phi$:

where $C$ is a relevant constant w.r.t. $x_{i}$ and $z_{i}$. According to the $LogSumExp$ trick, one can compute $logp_{\theta}(z_{i}|x_{i})$ by ignoring the above constant $C$:

where the optimum interpretation $z_{i}^{*}$ of the input region proposal $b_{i}$ is computed as $z_{i}^{*}=\operatorname*{arg\,max}_{z_{i}}logp_{\theta}(z_{i}|x_{i})$.

Moreover, a classical cross-entropy loss is employed in the relevant variational learning step to fit the above $p_{\theta}(z|x)$ with the ground-truth training samples:

where $\mathbb{L}(\theta)$ represents the variational learning objective, $c$ is the number of training images in a mini-batch, $\hat{z_{k}}$ is the ground-truth scene graph of the input image $\hat{x_{k}}$.

Finally, to better illustrate the proposed IWSL method, we summarize it in Algorithm 1. As a MFVB framework, the proposed IWSL method consists of two procedures: variational inference and variational learning. In particular, the variational inference procedure includes steps 3-7, while steps 8-9 represent the variational learning procedure. The temperature annealing process is accomplished in step 10. For computational efficiency, the number of samples $s$ in variational inference is often smaller than the one applied in variational learning.

Specifically, in variational inference, one first randomly initialize a categorical probability $\pi$ for a potential region proposal $b$, and then draw $s$ Gumbel noise samples $\sigma_{1},...,\sigma_{s}$ from a Gumbel noise distribution $u(\sigma)$. The output samples $z_{1},...,z_{s}$ are explicitly computed by feeding the above Gumbel noises $\sigma_{1},...,\sigma_{s}$ into a Gumbel-Softmax reparameterization function $g_{\pi}$. With the Equations (11), (12) and (16), one can easily compute the relevant importance weight $log\frac{s_{\theta}(x,z)}{q_{\pi}(z)}$ and approximate the corresponding importance weighted lower bound $\mathcal{L}_{s}$ via an Monte Carlo estimation. Finally, an entropic mirror descent (EMD) (which is introduced in the following subsection) method is applied to solve the resulting constrained variational inference task and obtain the optimum categorical probability $\pi^{*}$. In variational learning, according to Equation (17), one can first compute a surrogate logit $\phi$ based on the above updated optimum categorical probability $\pi^{*}$, and then obtain the resulting $logp_{\theta}(z|x)$ using the $LogSumExp$ trick. Finally, we compute the variational learning objective $\mathbb{L}(\theta)$ and then update the corresponding $\theta$ according to the stochastic gradient descent method.

With the above variational inference and learning procedures, in the training period, one can obtain the optimum $\theta^{*}$ and $\tau^{*}$. In the test period, $\theta$ and $\tau$ are fixed to $\theta^{*}$ and $\tau^{*}$, respectively. In particular, one discards the steps 9 and 10 in Algorithm 1, and generates the optimum interpretations $z^{*}=\operatorname*{arg\,max}_{z}logp_{\theta}(z|x)$.

As demonstrated in Equation (13), the variational inference procedure in the proposed method requires us to solve a constrained optimization problem. Specifically, to approximate the computationally intractable log partition function $logs_{\theta}(x_{i})$, one needs to maximize the $s$-sample importance weighted lower bound $\mathcal{L}_{s}^{i}$, subject to the constraint that the categorical probability $\pi_{i}$ resides in $(v-1)$-simplex.

Among the existing constrained optimization algorithms, entropic mirror descent (EMD) [25] method is chosen as the applied variational inference methodology, as demonstrated in Algorithm 2. Since the above constraint is a probability simplex, the negative entropy can naturally be employed as a specific function to construct the required Bregman distance [47]. Due to the utilization of the geometry of the optimization problem [48], EMD generally converges faster than the classical projected gradient descent methods [49].

The importance weighted lower bound becomes an unbiased estimator when the number of samples reaches infinity. However, it is impossible to achieve the above learning scenario in reality, due to the high computational complexity and the huge computational resources. In practice, for computational efficiency, the number of samples is often set to a relatively small number. To investigate the detection performance dependency of the proposed IWSL method on the number of samples, in this section, we choose three settings and compare their impact on the SGDet performance as shown in Table 4. We observe that the SGDet performance gradually increases with the number of samples. The performance gain obtained from a larger number of samples ($s>50$) is not that obvious, while its corresponding computation time increases dramatically. To pursue a balanced trade-off between the detection performance and the computational complexity, in this paper, the number of samples $s$ is set to $50$ in variational inference step.

To visually demonstrate the superiority of the proposed methods, in Fig. 3, we compare the visualization of the qualitative results of the ground-truth (GT), the baseline BGNN model, the proposed IWSL method and the derived IWSL+BA algorithm in the SGDet task. Compared with the baseline BGNN model, the proposed IWSL method is capable of detecting more informative $body$ and $tail$ predicates. For example, the proposed IWSL could detect an additional $<laying\;on>$ predicate for the top image, or an additional $<standing\;on>$ predicate for the middle image. Besides, the spatial informative predicates such as $<on\;back\;of>$ can also be detected. Moreover, the derived IWSL+BA method further improves the above capability. For example, it could detect an additional triplet $<cat\;in\;front\;of\;window>$ for the top image, or even a new reasonable triplet $<wing\;of\;airplane>$ (which is not included in the ground-truth scene graph GT) for the bottom image. In a word, compared with the baseline BGNN model, the scene graphs generated by the proposed IWSL method and the derived IWSL+BA algorithm are much closer to the ground-truth scene graph GT.