Relating by Contrasting: A Data-efficient
Framework for Multimodal DGMs

Although similar to the VAE LABEL:eq:elbo, our objective eq.˜2 directly maximises $p_{{\Theta}}\left(\bm{x},\bm{y}\right)$ in . $\mathcal{L}_{C}~$by itself is not a effective objective as in $\mathcal{L}_{C}~$minimises $p_{{\Theta}}\left(\bm{x},\bm{y}\right)$, which can overpower the effect of during training.

We intuit this phenomenon using a simple example in Figure˜4 with natural images, showing the log likelihood of $\log p(\bm{x},\bm{y})$ in column 2, 3, 4 (green) on a Gaussian distribution $\mathcal{N}((\bm{m_{x}},\bm{m_{y}});c)$, with the images in the first column of Figure˜4 as means and with constant variance $c$. While achieving high $\log p(\bm{x},\bm{y})$ requires matching $(\bm{m_{x}},\bm{m_{y}})$ (col 2), we see both unrelated digits (col 3) and noise (col 4) can lead to (comparatively) poor joint log likelihoods. This indicates that the generative model need not generate valid, unrelated images to minimise —generating noise would have roughly the same effect on log likelihood. As a result, the model can trivially minimise $\mathcal{L}_{C}~$by generating noise that minimises instead of accurate reconstruction that maximises .

This learning dynamic is verified empirically, as we show in Figure˜9 in Appendix˜D: optimising $\mathcal{L}_{C}~$by itself results in the loss approaching 0 within the first 100 iterations, and both and takes on extremely low values, resulting in a model that generates random noise.

To mitigate this issue, we need to ensure that minimising does not overpower maximising . We hence introduce a hyperparameter $\gamma$ on to upweight the maximisation of the marginal likelihood, with the final objective to minimise

We conduct ablation studies on the effect of $\gamma$ in Appendix˜E, noting that larger $\gamma$ encourages better quality of generation and more stable training in some cases, while models trained with smaller $\gamma$ are better at predicting “relatedness” between multimodal samples. We also note that optimising Equation˜3 maximises the pointwise mutual information $I(\bm{x},\bm{y})$; see Appendix˜B for a proof.

The dataset is designed to separate conceptual complexity, i.e. digit, from perceptual complexity, i.e. color, style, size. Each data pair contains 2 samples of the same digit, one from each dataset (see examples in Figure˜3(a)). We construct the dataset such that each instance from one dataset is paired with $30$ instances of the same digit from the other dataset. Although both datasets are simple and well-studied, the many-to-many pairing between samples creates matching of different writing styles vs. backgrounds and colors, making it a challenging multimodal dataset.

We also consider a more challenging language-vision multimodal dataset, Caltech-UCSD Birds (CUB) (Welinder et al., b; Reed et al., 2016). The dataset contains 11,788 photos of birds, paired with $10$ captions describing the bird’s physical characteristics, collected through Amazon Mechanical Turk (AMT). See CUB image-caption pair in Figure˜1.

Criterion: latent space factors into “shared” and “private” subspaces across modalities. We fit a linear classifier on the samples from $\bm{z}\sim q_{\Phi}(\bm{z}|\bm{x},\bm{y})$ to classify the information shared between the two modalities. For MNIST-SVHN, this can be the digit label as shown in §˜4.2 (right). We check if $\hat{l}_{z}$ is the same as the digit label of the original inputs $\bm{x}$ and $\bm{y}$, with the intuition that extracting the commonality between $\bm{x}$ and $\bm{y}$ from latent representation using a linear transform, supports the claim that the latent space has factored as desired.

Criterion: model generates paired samples that preserves the commonality observed in data. Again taking MNIST-SVHN as an example, this can be verified by taking pre-trained MNIST and SVHN digit classifiers and applying them on the multimodal observations generated from the same prior sample $z$. Coherence is computed by how often generations $\hat{\bm{x}}$ and $\hat{\bm{y}}$ classify to the same digit, i.e. whether $\hat{l}_{\hat{x}}=\hat{l}_{\hat{y}}$.

Criterion: model generates data in one modality conditioned on the other while preserving shared commonality. To compute cross coherence, we generate observations $\hat{\bm{x}}$ using latent from unimodal marginal posterior $\bm{z}\sim q_{\phi}(\bm{z}|\bm{y})$ and $\hat{\bm{y}}$ from $\bm{z}\sim q_{\phi}(\bm{z}|\bm{x})$. Similar to joint coherence, the criterion here is evaluated by predicting the label of the cross-generated samples $\hat{\bm{x}}$, $\hat{\bm{y}}$ using off-the-shelf MNIST and SVHN classifiers. In §˜4.2 (right), the cross coherence is the frequency of which $\hat{l}_{\hat{\bm{x}}}=l_{\bm{y}}$ and $\hat{l}_{\hat{\bm{y}}}=l_{\bm{x}}$.

Criterion: models learnt across multiple modalities should be no worse than those learnt from just one. For consistency, we evaluate this criterion from a coherence perspective. Given generations $\hat{\bm{x}}$ and $\hat{\bm{y}}$ from $\bm{z}\sim q_{\Phi}(\bm{z}|\bm{x},\bm{y})$, we again examine if generated and original labels match; i.e. if $\hat{l}_{\hat{\bm{x}}}=l_{\bm{y}}=\hat{l}_{\hat{\bm{y}}}=l_{\bm{x}}$.

See Appendix˜F for details on architecture. All quantitative results are reported over 5 runs. In addition to these quantitative metrics, we also showcase the qualitative results on both datasets in Appendix˜G and Appendix˜H.

See Table˜1 (top, 100% data used) for results on the full MNIST-SVHN dataset. Note that for MMVAE, since the joint posterior $q_{\Phi}$ is factorised as the mixture of unimodal posteriors $q_{\phi_{x}}$ and $q_{\phi_{y}}$, the model never directly takes sample from the explicit form of the joint posterior. Instead, it takes equal number of samples from each unimodal posteriors, reflective of the equal weighting of the mixture. As a result, it is not meaningful to compute synergy coherence for MMVAE as it is exactly the same as the coherence of any single-way generation.

From Table˜1 (top, 100% data used), we see that models trained on our contrastive objective (cI-<MODEL> and cC-<MODEL>) improves multimodal learning performance significantly for all three generative models evaluated on the metrics. The results showcase the robustness of our approach from the perspectives of modelling choice and metric of interests. In particular, note that for the best performing model MMVAE, using IWAE estimator for of Eq Equation˜3 (cI-MMVAE) yields slightly better results than CUBO (cC-MMVAE), while for the two other models the performance for different estimators are similar.

We also include qualitative results for MNIST-SVHN, including generative results, marginal likelihood table and diversity analysis in Appendix˜G.

Following Shi et al. (2019), for the images in CUB, we observe and generate in feature space instead of pixel space by preprocessing the images using a pre-trained ResNet-101 (He et al., 2016). A nearest-neighbour lookup among all the features in the test set is used to project the feature generations of the model back to image space. This helps circumvent CUB image complexities to some extent—as the primary goal here is to learn good models and representations of multimodal data, rather than a focus on pixel-level image quality of generations.

The metrics listed in §˜4.2 can also be applied to CUB with some modifications. Since bird-species classes are disjoint for the train and test sets, and as we show in Figure˜1 contains substantial in-class variance, it is not constructive to evaluate these metrics using bird categories as labels. In Shi et al. (2019), the authors propose to use Canonical Correlation Analysis (CCA)—used by Massiceti et al. as a reliable vision-language correlation baseline—to compute coherence scores between generated image-caption pairs; which we employ (i.e. (b), (c), (d) in §˜4.2) for CUB.

We show the results in Table˜2 (top, 100% data used). We see that our contrastive approach (both cI-<MODEL> and cC-<MODEL>) is even more effective on this challenging vision-language dataset, with significant improvements to the correlation of generated image-caption pairs. It is also on this more complicated dataset where the advantages of using the stable, low-variance IWAE estimator are highlighted — for both MVAE and JMVAE, the contrastive objective with CUBO estimator suffers from numerical stability issues, yielding close-to-zero correlations for all metrics in Table˜2. Results for these models are therefore omitted.

We also show the qualitative results for CUB in Appendix˜H.

As showing in Figure˜7, we first construct a full dataset by randomly matching instances in MNIST and SVHN, and denote the related pairs by $F_{r}$ (full, related). We further assume access to only $n$% of $F_{r}$, denoted as $S_{r}$ (small, related), and denote the rest as $F_{m}$, containing a mix of related and unrelated pairs. Next, we train a generative model $g$ on $S_{r}$. To find a relatedness threshold, we construct a small, mixed dataset $S_{m}$ by randomly matching samples across modalities in $S_{r}$. Given relatedness ground-truth for $S_{m}$, we can compute the PMI $I(\bm{x},\bm{y})=\log p_{\Theta}(\bm{x},\bm{y})-\log p_{\theta_{x}}(\bm{x})p_{\theta_{y}}(\bm{y})$ for all pairs $(\bm{x},\bm{y})$ in $S_{m}$ and estimate an optimal threshold. This threshold can now be applied to the full, mixed dataset $F_{m}$ to identify related pairs giving us a new related dataset $F_{\hat{r}}$, which can be used to further improve the performance of the generative model $g$.

In Figure˜8 (a-e), we plot the performance of baseline MMVAE (blue) and contrastive MMVAE with IWAE (cI-MMVAE, red) and CUBO (cC-MMVAE, yellow) estimators for term of Eq Equation˜3, trained with (solid curves) and without (dotted curves) label propagation. Here, the x-axis is the proportion in size of $S_{r}$ to $F_{r}$, i.e. the percentage of related data used to pretrain the generative model before label propagation. We compare these results to MMVAE trained on all related data $F_{r}$ (cyan horizontal line) as a “best case scenario” of these training regimes.

Clearly, label propagation using a contrastive model with IWAE estimator is helpful, and in general the improvement is greater when less data is available; Figure˜8 also shows that when $S_{r}$ is 10% of $F_{r}$, cI-MMVAE is competitive with the performance of baseline MMVAE trained on $F_{r}$.

For baselines MMVAE and cC-MMVAE however, label propagation hurts performance no matter the size of $S_{r}$, as shown by the blue and yellow curves in Figure˜8 (a-e). The advantages of cI-MMVAE is further demonstrated in Figure˜8(f), where we compute the precision, recall, and $F_{1}$ score of relatedness prediction on $F_{m}$, for models trained on 10% of all related data. We also compare to a simple label-propagation baseline, where the relatedness of $F_{m}$ is predicted using a siamese network (Hadsell et al., 2006) trained on the same 10% dataset. Notably, while the Siamese baseline in Figure˜8(f) is a competitive predictor of relatedness, cI-MMVAE has the highest $F_{1}$ score amongst all four, making it the most reliable indicator of relatedness. Beyond that, note that with the contrastive MMVAE, relatedness can be predicted without additional training and only requires a simple threshold computation directly computed using the generative model. The fact that the cI-MMVAE’s relatedness-prediction performance is the only one that matches the Siamese baseline strongly supports the view that the contrastive loss encourages generative models to utilise and better learn the relatedness between multimodal pairs; in addition, the poor performance of cC-MMVAE shows that the advantages of having a bounded estimation to the contrastive objective by using an upper-bound for is overshadowed by the high bias of CUBO, and that one may benefit more from choosing a low variance lower-bound like IWAE.