L2C: Describing Visual Differences
Needs Semantic Understanding of Individuals

To extract semantic visual features, we utilize pre-trained fully convolutional networks (FCN) (Long et al., 2015) with ResNet-101 as the backbone. An image $\mathcal{I}$ is fed into the ResNet backbone to produce a feature map $\mathcal{F}$ $\in\mathbb{R}^{D\times H\times W}$, which is then forwarded into an FCN head that generates a binary segmentation mask $B$ for the bird class. However, the shapes of these masks are variable for each image, and simple pooling methods such as average pooling and max pooling would lose some information of spatial relations within the mask.

To address this issue and enable efficient aggregation over the area of interest (the masked area), we add a module after the ResNet to cluster each pixel within the mask into $K$ classes. Feature map $\mathcal{F}$ is forwarded through this pooling module to obtain a confidence map $\mathcal{C}$ $\in\mathbb{R}^{K\times H\times W}$, whose entry at each pixel is a $K$-dimensional vector that represents the probability distribution of $K$ classes.

Then a set of nodes $V=\{v_{1},...,v_{K}\},v_{k}\in\mathbb{R}^{D}$ is constructed as following:

$$v_{k}=\sum_{i,j}\mathcal{F}\odot\mathcal{B}\odot\mathcal{C}_{k}$$

(1)

where $i$=$1,...H,$ $j$=$1,...,W,$, $\mathcal{C}_{k}$ is the $k$-th probability map and $\odot$ denotes element-wise multiplication.

To enforce local smoothness, i.e., pixels in a neighborhood are more likely belong to one class, we employ total variation norm as a regularization term:

$$\mathcal{L}_{TV}=\sum_{i,j}|C_{i+1,j}-C{i,j}|+|C_{i,j+1}-C{i,j}|$$

(2)

Inspired by recent advances in visual reasoning and graph neural networks  (Chen et al., 2018; Li et al., 2019), we introduce a relational reasoning module to enhance the semantic representation of each image. A fully-connected visual semantic graph $G=(V,E)$ is built, where $V$ is the set of nodes, each containing a regional feature, and $E$ is constructed by measuring the pairwise affinity between each two nodes $v_{i},v_{j}$ in a latent space.

$$A(v_{i},v_{j})=(W_{i}v_{i})^{T}(W_{j}v_{j})$$

(3)

where $W_{i},W_{j}$ are learnable matrices, and $A$ is the constructed adjacency matrix.

We apply Graph Convolutional Networks (GCN)  (Kipf and Welling, 2016) to perform reasoning on the graph. After the GCN module, the output $V^{o}=\{v_{1}^{o},...,v_{K}^{o}\},v_{k}^{o}\in\mathbb{R}^{D}$ will be a relationship enhanced representation of a bird. For the visual comparison task, we compute the difference of each two visual nodes from two sets, denoted as $V^{g}_{diff}=\{v_{diff,1}^{o},...,v_{diff,K}^{o}\},v_{diff,k}^{o}=v_{k,1}^{o}-v_{k,2}^{o}\in\mathbb{R}^{D}$.

After obtaining relation-enhanced semantic features, we use a Long Short-Term Memory (LSTM)  (Hochreiter and Schmidhuber, 1997) to generate captions. As discussed in Section 1, semantic understanding of each image is key to solve the task. However, there is no single dataset that contains both visual comparison and single-image annotations. Hence, we leverage two datasets from similar domains to facilitate training. One is for visual comparison, and the other is for single-image captioning. Alternate training is utilized such that for each iteration, two mini-batches of images from both datasets are sampled independently and fed into the encoder to obtain visual representations $V^{o}$ (for single-image captioning) or $V^{o}_{diff}$ (for visual comparison).

The LSTM takes $V^{o}$ or $V^{o}_{diff}$ with previous output word embedding $y_{t-1}$ as input, updates the hidden state from $h_{t-1}$ to $h_{t}$, and predicts the word for the next time step. The generation process of bi-image comparison is learned by maximizing the log-likelihood of the predicted output sentence. The loss function is defined as follows:

$$\mathcal{L}_{diff}=-\sum_{t}{\log P(y_{t}|y_{1:t-1},V^{o}_{diff})}$$

(4)

Similar loss is applied for learning single-image captioning:

$$\mathcal{L}_{single}=-\sum_{t}{\log P(y_{t}|y_{1:t-1},V^{o})}$$

(5)

Overall, the model is optimized with a mixture of cross-entropy losses and total variation loss:

$$\begin{split}\mathcal{L}_{loss}=\mathcal{L}_{diff}+\mathcal{L}_{single}+\lambda\mathcal{L}_{TV}\end{split}$$

(6)

where $\lambda$ is an adaptive factor that weighs the total variation loss.

As shown in Table 1, first, L2C[B2W] (training with visual comparison task only) outperforms baseline methods on BLEU-4 and ROUGE-L. Previous approaches and architectures failed to bring superior results by directly modeling the visual relationship on ResNet features. Second, joint learning with a single-image caption L2C[B2W+CUB] can help improve the ability of semantic understanding, thus, the overall performance of the model. Finally, our method also has a smaller gap between validation and test set compared to neural naturalist, indicating its potential capability to generalize for unseen samples.

To fully evaluate our model, we conduct a pairwise human evaluation on Amazon Mechanical Turk with 100 image pairs randomly sampled from the test set, each sample was assigned to 5 workers to eliminate human variance. Following Wang et al. (2018), for each image pair, workers are presented with two paragraphs from different models and asked to choose the better one based on text quality²²2We instruct the annotators to consider two perspectives, relevance (the text describes the context of two images) and expressiveness (grammatically and semantically correct).. As shown in Table 2, L2C outperforms CNN+LSTM, which is consistent with automatic metrics.