What Remains of Visual Semantic Embeddings

Let $D_{S}=\{(x,y)\mid x\in X_{S},y\in Y_{S}\}$ be a set of training examples consisting of seen images from $X_{S}$ and seen class labels from $Y_{S}$. Let $D_{U}=\{(x,y)\mid x\in X_{U},y\in Y_{U}\}$ be a set of testing examples. $Y_{U}$ represents the unseen set of class labels, which is disjoint from $Y_{S}$. The task of ZSL is to learn a classifier, $X\rightarrow Y$, where $X$ is the union set of $X_{S}$ and $X_{U}$ and $Y$ is the union set of $Y_{S}$ and $Y_{U}$.

The idea of word embedding methods is derived from the distributional hypothesis in linguistics: words that are used and occur in the same contexts tend to purport similar meanings [10]. Word representations learned in an unsupervised manner from the contextual relationship of words, or the co-occurrence statistics of words, in large text corpora are the main source of high-level semantic knowledge in most ZSL approaches. In ZSL these word embedding models provide a mapping, $Y\rightarrow W$, from class labels to high dimensional word vectors. We denote the word vector corresponding to a label $y$ as $w(y)$. Cosine similarity is usually used to predict word similarity. For a pair of word representations $w_{i}$ and $w_{j}$, we have:

$$\mathrm{similarity}(w_{i},w_{j})=\frac{w_{i}\top w_{j}}{\left\|w_{i}\right\|\left\|w_{j}\right\|}\enspace.$$

Previous ZSL models begin with training a classifier on the training set. This step leaks specific semantic information into the feature extractor component of a ZSL system. However, this step can not guarantee that we can obtain linear separable novel image features, therefore, it is very difficult to figure out whether the feature extractor or the downstream visual semantic embedding model affect the generalization performance. Contrastive learning (CL) provides a framework to learn representations of identity by pushing apart two views of different objects and bringing together two views of same objects in a representation space [4]. Recent works show that CL can generate linear separable image features without any supervised label information [26, 35, 12, 3]. Given an anchor variables from the first view $v_{1,i}$, CL aims to score the correct positive variable from the second view $v_{2,i}\sim p(v_{2}\mid v_{1,i})$ higher compared to a set of $K$ negatives $v_{2,k}\sim p(v_{2})$. A popular approach is to utilise the InfoNCE loss [26]:

$$L_{NCE}=-\mathbf{E}\left[\log\frac{\exp{h(f(v_{1,i}),f(v_{2,i}))}}{\sum_{j=1}^{K}\exp{h(f(v_{1,i}),f(v_{2,j}))}}\right]$$

The function $h$ represents a critic head and the function $f$ is a shared encoder, which extracts view-invariant visual representations This way of feature learning is natural and powerful. This allows the step of building the visual semantic embedding model to be the only stage to align visual information and distributed linguistic representations. When we freeze the visual encoder, we can evaluate what mechanism performs better on semantic understanding.

Graph convolutional networks (GCNs) [19] allow local graph operators to share the statistical strength between word vectors of classes, which is considered as a tool to utilize semantic information hidden in the WordNet hierarchy.

Given a graph with $N$ classes and a $d$ word embedding per class, $\mat{W}$ is the $N\times d$ word embedding matrix. We define a symmetric adjacency matrix $\mat{A}$ and a degree matrix $\mat{D}$ to represent the connections between the classes in the WordNet. The propagation rule to perform convolutions on the graph is defined as:

$\displaystyle\mat{H}_{l+1}$

$\displaystyle=\sigma(\mat{D}^{-1}\mat{A}\mat{H}_{l}\mat{\Theta}_{l}),$

$\displaystyle\mat{H}_{0}$

$\displaystyle=\mat{W}$

Here $\mat{H}_{l}$ is the input at each layer of the GCN and $\mat{H}_{l+1}$ is the output. $\sigma$ is the activation function and $\mat{\Theta}_{l}$ is the learnable parameter matrix at the layer $l$. We define a two layer GCN as $g$.

The Poincaré embedding model is another natural technique to capture hierarchical information in the WordNet hierarchy. As it is not pre-trained on a large text corpus, each node in the Poincaré embedding model does not contain any word co-occurrence statistics. Poincaré embeddings preserve the distances between the nodes on the graph approximately, which can be used to compute semantic similarities. Given two Poincaré embeddings $p_{i}$ and $p_{j}$, we have the distance:

$$d(p_{i},p_{j})=\cosh^{-1}\left(1+2\frac{\left\|p_{i}-p_{j}\right\|^{2}}{\left\|1-p_{i}\right\|^{2}+\left\|1-p_{j}\right\|^{2}}\right)\;.$$

Two kinds of transformation are useful in a hyperbolic space. The mapping $e$ to project a vector $v$ in Euclidean space to hyperbolic space is defined as:

$$p=e(v)=\tanh(\left\|v\right\|)\frac{v}{\left\|v\right\|}$$

The Mobius multiplication $m$ is defined as:

$$m(x)=\tanh\left(\frac{\left\|mx\right\|}{\left\|x\right\|}\tanh^{-1}(\left\|x\right\|)\right)\frac{mx}{\left\|x\right\|}$$

The standard data split in the ImageNet has its own structural flaws. Test classes are the hypernym and hyponym of training classes. On the other hand, testing images contain many inconsistencies, it is hard to ensure images with high quality are tested. While tieredImageNet is a large subset of ILSVRC 2012 1k, in which classes are nodes of a tree of height 13 covering 608 classes. To build a ZSL benchmark, we combine the training classes and validation classes of tieredImageNet as the seen set, and we keep its own testing classes. In our ZSL setting, 448 classes are in the seen set, and 160 classes are in the testing set. Images in the training set of ILSVRC 2012 1k with the seen labels are used to train a visual semantic embedding model. Images in the validation set of ILSVRC 2012 1k with the seen labels are used to evaluate embedding performance. Images in the validation set of ILSVRC 2012 1k with the unseen labels are used to evaluate ZSL performance.

For pre-trained image encoder, we use a ResNet-50 trained for 800 epochs on the ILSVRC 2012 1k with InfoMin CL Algorithm [36] ¹¹1https://github.com/HobbitLong/PyContrast. With a linear probe, this image encoder shows $73.0\%$ classification accuracy on the ILSVRC 2012 1k task. We choose 300 dimensional GloVe²²2https://nlp.stanford.edu/projects/GloVe/ with 6B tokens as the pre-trained word vectors. For each class label, we average all the GloVe vectors of its synonyms. We train a Poincaré embedding model with gensim ³³3https://radimrehurek.com/gensim/ from the tree of tieredImageNet classes. We choose a 100 dimensional hyperbolic space for this model.

The transformer in DeVISE is a two-layer multilayer perceptron (MLP) with 512 hidden neurons. The feature encoder and decoder in PrVISE are both two-layer MLPs with 512 hidden neurons. The adaptive prior distribution in PrVISE is 300 dimensional, learnt by a two-layer MLP. In GrVISE, a two-layer GCN is used to predict weights of the visual classifier. And in HrVISE, a two-layer MLP is modified to implement Mobius multiplication twice.

All the visual semantic embedding models are trained with encoder freezing. The Adam optimizer [17] with learning rate 1e-4 is used in all the models to optimize the loss function. All the models are trained for 200 epochs and mini-batches of size 256.

We consider different kinds of measures to answer the questions provided at the beginning of this section.

To demonstrate the performance of visual semantic embeddings, we compute the embedding hit@k metrics – the percentage of test images in the seen set for which the model returns the one true seen label in its top k predictions. For the ZSL task, we consider the generalized ZSL. Therefore, we compute the ZSL-S hit@k ZSL-U hit@k metrics. Here S represents test images in the seen set, while U represents test images in the unseen set.

Inspired by the average hierarchical distance of a mistake [1], which rethinks whether we should treat all classes other than the “true” label as equally wrong, we also employ the average similarity of a mistake (avg.sim@k) and the average similarity distance of a mistake (avg.sim.dis@k) as metrics to demonstrate whether a visual semantic embedding model remains the properties of a word embedding space. To compute these two metrics, we firstly compute the cosine similarity matrix of the GloVe vectors. Then we sort these similarities and get the distance of each pair of labels. For the data instances which the model does not return the true labels, we average the similarities and distances between the predicted labels and the true labels in their top $k$ predictions. In the ZSL part, we also employ ZSL-S avg.sim@k, ZSL-U avg.sim@k, ZSL-S avg.sim.dis@k and ZSL-U avg.sim.dis@k. The difference is we build the similarity matrices with the union set of training labels and testing labels.

We think the new metrics can help us figure out the effectiveness of semantic alignment in the ZSL task, as Roads and Love [29] employed the Spearman correlation between the upper diagonal portion of two similarity matrices in two conceptual systems.

To answer this question, we should consider the paradigms purely dealing with information from word embeddings. In the following, we analyse the performance of DeVISE and PrVISE. As a comparison, we implement a linear classifier (LP) on top of the frozen representation encoded by the InfoMin encoder. Although the linear probe can not be used in the ZSL task, it still gives insights about how image representations are distributed with equally weighted labels.

The Table 1 shows results to evaluate visual semantic embedding performance. We highlight that only seen classes in the testing set is used in this part. A linear layer classifier performs better than the DeVISE and the PrVISE models on the flat hit metrics. Although there are no distributed semantics leaked to the LP, it still shows the highest semantic similarities for the wrong classified data.

DeVISE performs slightly worse than the linear classifier. There is no evidence to prove that pre-trained word embeddings help neural networks make a more reasonable decision at the semantic level, as random embeddings still work well on the classification task [7, 1].

The generative model PrVISE shows the worst performance. Unlike the success on medium datasets with human designed attributes, the VAE-based model struggles to classify images by comparing the KL-divergence between latent distributions. This indicates a big performance gap exists when the latent space can not be disentangled. However, the PrVISE model “makes better semantic mistakes”. Focusing on the difference of average similarities on the top 1 and top 5 predictions, these models do not prefer to choose candidate labels in the semantic synonyms of true labels. The PrVISE model does decrease the semantic distance of a mistake on the top 5 predictions. Although, it is still very hard to answer the question “Is visual similarity correlated to semantic similarity?” [6], it seems that the PrVISE model exchanges the embedding accuracy with the semantic accuracy. When we evaluate the performance of visual semantic embedding with extended labels, this phenomenon is more clear. DeVISE and PrVISE can deal with a flexible number of classes. When the number of label classes increases, the embedding performance drops (see Table 2), but the PrVISE model still “makes better semantic mistakes”. For unseen classes (Table 3), PrVISE performs worse than DeVISE. Notably, the performance in this work with contrastive learning pre-training reaches the same order of magnitude as the traditional setting, even though our data split is more difficult.

We test the performance of GrVISE and HyVISE to demonstrate how a semantic graph affects the way a model aligns visual semantics. We discuss GrVISE and HyVISE respectively.

GrVISE is designed to predict the parameters of a linear image classifier with the help of GCN. We set a two-layer MLP as a comparison. In previous work using a GCN based model [37], parameters in the last layer of a pre-trained classifier are considered to be naturally normalized. However, we notice that the parameters in a pre-trained linear probe on a freezing encoder are not normalized. Therefore, we normalize the parameters of the linear probe. Note that the normalized parameters still work well the original classification task.

We observe that an MLP performs better on parameter prediction. Therefore, it achieves better embedding performance for seen classes. While a GCN performs better to predict unseen labels, this demonstrates that a GCN does learn a bit of structural information from a hand-crafted semantic graph.

However, the GrVISE does not organize visual semantic information better than the DeVISE. We argue that the process of learning a parameter prediction task can gradually acquire semantic information for unseen classes. We train a linear probe on both seen and unseen classes. Then we train a GCN and an MLP respectively to predict the parameters of seen classes. Note that the probe can achieve $79.10\%$ classification accuracy on unseen classes.

In Figure 4, we show that the GrVISE can not precisely learn unseen parameters. The dashed lines representing the prediction error for unseen class parameters are observed stopping decreasing at the early epoch. The WordNet structure does not narrow the semantic gap between seen classes and unseen classes.

Finally, we test the HyVISE, which can been seen as the DeVISE in Riemannian geometry. We get 13.91% embedding hit@1, 13.84% ZSL-S hit@1 and 0.219% ZSL-U hiy@1. The results indicate that a Poincaré visual semantic embedding model does not work well on a complicate ZSL data split, when the split is based on a sub tree of the WordNet.

In summary, we find that the DeVISE still works robustly on a new ZSL class split, which needs high-level semantic understanding. Compared with a linear classifier, DeVISE does not show that it has performed semantic inference using the distributed semantics. The generative model PrVISE performs slightly worse than the DeVISE, but it shows the advantage of predicting better mistakes, which are semantically similar to the ground truth labels. The graph-based models GrVISE and HyVISE lose their advantages shown the standard splits. The results demonstrate that recent semantic alignment techniques could overfit to the original flawed problem, and they also indicate we need a deeper discussion on the ZSL task in the real world.