On The Multi-View Information Bottleneck Representation

Recently, learning from multi-view data has drawn significant interest in the machine learning and data science community (e.g., [1, 2, 3, 4, 5]). In this context, a view of data is a description or observation about the source. For example, an object can be described in words or images. It is natural to expect learning from multi-view data to improve performance[6].

The challenges in multi-view learning are two-folded: First, as one can naively combine all views of observations to form one giant view which loses no information contained within them but would suffer from exponential growth of the dimensionality of the merged observation, we call this the performance-complexity trade-off. Second, following this, if one instead opts to extract either view-shared or view-specific relevant features from each view, then heterogeneous forms of observations (e.g., audio and images) would make it difficult to learn low-complexity and meaningful representations with a unified method. In other words, the amount of representation overlap across all view observations is important for efficient multi-view learning.

In addressing the challenges, several recent works have attempted to apply the IB [7] principle to multi-view learning for it matches the objective well, that is, trading-off relevance and complexity in extracting both view-shared and view-specific features [8, 9, 10, 11]. This generalization is known as the multi-view IB (MvIB) which is known to be a special case of the multi-terminal remote source coding problems with logarithmic loss. The logarithmic loss corresponds to soft reconstruction where the likelihood of all possible outcomes is received in contrast to a reconstructed symbol in conventional source coding problems [12, 13, 14, 15, 16].

In literature, the achievable region for the remote source coding problem is characterized in [17] for discrete cases, and recently, jointly Gaussian cases as well [13]. Along with the characterization, a variety of variational inference-based algorithms are proposed[14, 15]. This type of algorithms introduces extra variational variables to facilitate the optimization as by fixing one of the two sets of variables, the overall objective function is convex w.r.t. to the other set of variables. Then optimize in this alternating fashion, the convergence is assured.

Extending this line of research, our approach is rooted in a top-down information-theoretic formulation closely related to the optimal characterization of MvIB. Moreover, contrary to [11] which relies on black-box deep neural networks, we propose two constructive information theoretic formulations with performance comparable to that of the optimal joint view approach. The two approaches are motivated by two extreme multi-view learning scenarios: The first is characterized by a significant representation overlap between the different views which favors our consensus-complement two stage formulation, whereas the second extreme scenario is characterized by a minimal representation overlap leading to our incremental update approach.

Different from existing variational inference-based algorithms that avoid dealing with the non-convexity of the overall objective function, in both of the proposed methods, we adopt the non-convex consensus ADMM as the main tool in deriving our solvers [18, 19, 20]. These new solvers can, therefore, be viewed as generalizations of our earlier work on the single view ADMM algorithm [21]. More specifically, in the consensus-complement version, we separate the proposed Lagrangian into consensus and complement sub-objective functions and then proceed to solve the optimization problem in two steps. The new ADMM solver can hence efficiently form a consensus representation in large-scale multi-view learning problems with significantly lower dimensions compared to joint-view approaches. The same intuition is applied to the increment update approach as detailed in the sequel. Finally, we prove the linear rate of convergence of our solvers under significantly milder constraints; as compared with earlier convergence results on this class of solvers [22, 23, 24]. More specifically, we relax the need for a strongly convex sub-objective function and, moreover, establish the linear rate of convergence around a local optimal point in each case.

Given $V$ views with observations $\{X^{(i)}\}_{i=1}^{V}$ generated from a target variable $Y$, we aim to find a set of individual representations $\{Z\}$ that is most compressed w.r.t. the individual-view observations $X^{(i)},\forall i\in[V]$ but at the same time maximize their relevance toward the target $Y$ through $X^{(i)}$.

Using a Lagrangian multiplier formulation, the problem can be casted as:

where $\{X\}$ denotes the set of all $V$-views of observations and $\{Z\}$ is the set of representations to be designed. Note that if the observations are combined in this manner and treated as one view, the above reduces to the standard IB and can be solved with any existing single-view solver. However, combining all the observations in one giant view will result in an exponential increase in complexity (curse of dimensionality).

A basic assumption in the multiview learning literature is the conditional independence [25, 9], where the observations of all views $\{X^{(i)}\}_{i=1}^{V}$ are independent given the target variable $Y$. That is, $p(\{x\}|y)=\prod_{i=1}^{V}p(x^{(i)}|y)$. In the next two sections, we use this conditional independence assumption while constraining the set of allowable latent representations $\{Z\}$ to develop our two novel information-theoretic formulations of the Multi-view IB (MvIB) problem.

We propose two new information-theoretic formulations of MvIB and develop optimal-bound achieving algorithms that are in parallel to existing solvers [15, 26, 14]; our main results are the convergence proofs for the proposed two algorithms. The convergence analysis goes beyond the MvIB and the recent non-convex multi-block ADMM convergence results as we further show that strong convexity on $\{F_{i}\}_{i=1}^{V}$ is not necessary for proving convergence [24]. This new result connects our analysis to a more general class of functions that can be solved with multi-block non-convex consensus ADMM. For simplicity we denote the collective point at step $k$ as $w^{k}:=(\{p_{i}^{k}\},\{\nu^{k}_{i}\},q^{k})$, $\mathcal{L}_{c}^{k}:=\mathcal{L}_{c}(\{p_{i}^{k}\},\{\nu_{i}^{k}\},q^{k})$ as the function value evaluated with $w^{k}$ and $\mu_{B},\lambda_{B}$ denote the smallest and largest singular value of a linear operator $B$ respectively,

As a remark, if the minimum element of a probability vector is bounded away from zero by a constant $\xi>0$, a commonly adopted smoothness condition in density and entropy estimation research [28], the sub-objectives $\{F_{i}\}_{i=1}^{V}$ and $G$ can be shown to be Lipschitz continuous and smooth. Furthermore, under smoothness conditions, $G$ is a weakly convex function w.r.t. $q$ (Lemma 2).

From Theorem 1, the consensus-complement algorithm is convergent since the complement step is a special case of the algorithm (11) with $V=1$ while treating $p(z_{c}|x^{(i)})$ as an additional prior probability. The incremental algorithm is convergent following the same reason.

We evaluate the proposed two approaches for two-view, synthetic distributions. For simplicity, we denote the consensus-complement approach as Cons-Cmpl while the incremental update approach as Increment.

We simulate a classification task and compare the performance of the two proposed approaches to joint-view/single-view IB solvers [29], which are served as references for the best- and worst-case performance, and the state-of-the-art deep neural network-based method (DeepMvIB [14, 11]), with two layers of $4$-neuron, fully connected weights plus ReLU activation for each view. Given (19), we randomly sample $10000$ pairs of outcomes $(y,x^{(1)},x^{(2)})$ as testing data. Then we run the algorithms, sweeping through a range of $\gamma\in[0.1,0.7]$ and record the best accuracy from $50$ trials per $\gamma$. We use Bayes’ decoder to predict the testing data, where we perform inverse transform sampling for the cumulative distribution of the decoders to obtain $\hat{y}$ for each pair of $(x^{(1)},x^{(2)})$. The data-generating distribution is:

with $P(Y)=\begin{bmatrix}0.5&0.5\end{bmatrix}^{T}$. The result is shown in Figure 1a. The dimension of each of $Z_{c},Z_{e}^{(2)},Z^{(2)}$ is $2$, and $3$ for each of $Z_{e}^{(1)},Z^{(1)}$. Clearly, the two proposed approaches can achieve comparable performance to that of the joint-view IB solver and outperform the deepMvIB over the range of $\gamma$ we simulated. Interestingly, Cons-Cmpl outperforms Increment in the best accuracy. This might be due to the abundance of representation overlap. To better investigate this observation, we further consider a different distribution with dimensions of all representations $|\mathcal{Z}_{c}|=|\mathcal{Z}_{e}^{(i)}|=|\mathcal{Z}^{(i)}|=3,\forall i\in\{1,2\}$:

Observe that for each view in (20), there is one class ($y_{1}$ in view $1$ an $y_{2}$ in view $2$), that is easy to infer through $X^{(i)},i\in\{1,2\}$ while the remaining two are ambiguous. This results in low representation overlap and consensus is therefore difficult to form. In Figure 1b we examine the components of the relevance rate $I(\{Z\};Y)$ where the Sum is: $I(Z_{c};Y)+\sum_{i=1}^{2}I(Z_{e}^{(i)};Y|Z_{c}))$ for Cons-Cmpl and $I(Z^{(1)};Y)+I(Z^{(2)};Y|Z^{(1)})$ for Increment. Step 1 indicates $I(Z_{c};Y)$ in Cons-Cmpl, and $I(Z^{(1)};Y)$ in Increment. Observe that there is almost no increase in $I(Z_{c};Y)$ over varying $\gamma$, and that Increment has a greater relevance rate than Cons-Cmpl when $\gamma<0.4$. Since it is known that the high relevance rate is related to high prediction accuracy [30], this example favors the Increment approach as it is designed to increase the overall relevance rate view-by-view.

Lastly, we can compare the complexity of the two approaches in terms of the number of dimensions for the primal variables. For simplicity, let $|X|=|X^{(i)}|$,$|Z|=|Z_{c}|=|Z_{e}|=|Z^{(i)}|$. For Cons-Cmpl, the number of dimensions for the variables scales as $\mathcal{O}(V|X||Z|^{2})$ while for Increment, it grows as $\mathcal{O}(|X||Z|^{V})$. The two methods both improve over the joint view as their complexity values scale as $\mathcal{O}(|X|^{V}|Z|)$ and $|Z|\ll|X|$ in general. Remarkably, the complexity for Cons-Cmpl scales linearly in the number of views $V$ while we get an exponential growth with factor $|Z|$ for Increment.

In this work, we propose two new information-theoretic formulations of MvIB and develop new optimal bound-achieving algorithms based on non-convex consensus ADMM, which are in parallel to existing solvers. We proposed two algorithms to solve the two forms respectively and prove their convergence and linear rates. Empirically, they achieve comparable performance to joint-view benchmarks and outperforms state-of-the-art deep neural networks-based approaches in some synthetic datasets. For future works, we plan to evaluate the two methods on available multiview datasets [31, 32] and generalize the proposed MvIB framework to continuous distributions [33].