Self-supervised Multi-view Learning for Disentangled Music Audio Representations

SSL uses pretext tasks to uncover patterns from unlabeled data. In single-view SSL, a model learns from one perspective of the input via information restoration [1]. Multi-view SSL, however, utilizes distinct views to generate supervision, assuming shared information across views suffices for downstream tasks [2]. These methods align and contrast information across views for learning.

Recent multi-view SSL studies use contrastive learning to treat audio segments or augmentations as transformed views [3, 4]. However, these approaches neglect the intrinsic structure of music audio, entangling representations with attributes like timbre, frequency, and tude. They also focus on shared information across views, missing task-relevant, view-specific details.

Previous research has primarily focused on separating pitch and timbre to encode disentangled music representations, often designing dedicated encoders for each attribute. Some approaches train generative models with explicit supervision on pitch and timbre latents [5], while others reduce supervision by applying auxiliary metric-based regularization in the latent spaces [6, 7, 8, 9].

We propose a novel self-supervised multi-view learning framework for music audio, inspired by multi-view/multimodal disentanglement [10, 11], which explicitly separates shared and private representations. This approach preserves the uniqueness of each view while capturing common latent factors. As a case study, we tackle music audio disentanglement from a multi-view learning perspective with self-supervision. We validate our method using Syntone [12], a dataset with controlled variations in music attributes such as timbre and frequency.

Our method is shown in Figure LABEL:fig:block_diag for the case of two views. The model receives as input a pair of normalized log mel spectrograms, denoted as $(x_{1},x_{2})$, which are characterized by identical timbre (waveform class) but distinct frequency. Factorized latents $z_{1}\sim q_{\phi}(z|x_{1})$ and $z_{2}\sim q_{\phi}(z|x_{2})$ are inferred from a shared encoder $q$ parameterized by $\phi$. To separate the private and shared spaces, we split the latents into $z_{1}=[z_{p1},z_{s1}]$ and $z_{2}=[z_{p2},z_{s2}]$, where $z_{p1},z_{p2}\in\mathbb{R}^{D_{p}}$ are the private latents of each view and $z_{s1},z_{s2}\in\mathbb{R}^{D_{s}}$ correspond to the private-shared latents. Following [13], we use a PoE-based consistency model to approximate the true posterior $p(z_{s}|x_{1},x_{2})$ so that we effectively make $z_{s1}=z_{s2}=z_{s}$. For all the continuous latent variable $z$, we assume $p(\mathbf{z})=\mathcal{N}(0,I)$.

We set the latent dimensions to be $D_{p}=D_{s}=8$ and sample from the parameterized distributions to arrive at three $8$-dimensional latent variables: $\mathbf{z_{p1}}$, $\mathbf{z_{p2}}$, and $\mathbf{z_{s}}$ corresponding to the $x_{1}$-private, $x_{2}$-private, and shared representations, respectively. We concatenate each private latent $z_{pi,i\in\{1,2\}}$ with the shared latent $z_{s}$ as input to the shared decoder $p_{\theta}$ parameterized by $\theta$.

Following DMVAE [10] and $\beta$-TCVAE [14], the tractable evidence lower bound of our model assumes the form for each sample pair $x$ and $N$ views of the data:

The objective to maximize is the sum of the reconstruction accuracy compensated by the KL-Divergence for each view $i$. Particularly, each KL term in Eq (2) is decomposed into mutual information $MI(x_{i};z)$, total correlation $TC(z;\prod_{k}q_{\phi}(z_{k}))$, and dimension-wise KL $DKL(q_{\phi},p)$, with $\alpha$, $\beta$, and $\gamma$ as penalty weights respectively.

Data Synthesis: We generate a controlled dataset of $32,000$ 1-second audio samples and corresponding normalized log-mel spectrograms with varying timbre and frequency factors, similar to the data generation of SynTone [12]. We randomly choose pairs of log-mel spectrograms with the same timbre but with different frequencies to incentivize our model to learn shared timbre information and varying frequency information.

In Figure 1, we show that our model is able to clearly disentangle frequency information into the private subspace and timbre information into the shared in terms of the mutual information between factors and latent dimensions. This observation supports our hypothesis that a multi-view framework combined with a paired dataset that contains shared and private information should be able to learn separate embedding subspaces pertaining to different factors.

Our downstream classification performance in Table 1 shows similar trends to the above: when using the private latent or a combination of the private and shared, our model performs very well at predicting frequency, and when using the shared latent or combination of both, we are able to classify timbre successfully.

We also perform an ablation of weights of the decomposed KL-Divergence terms in Eq. 2 for $\alpha$, $\beta$, and $\gamma$. Surprisingly, we found that higher weights of $\alpha$ (MI) and $\beta$ (TC) negatively affect private-shared disentanglement, while increasing $\gamma$ as a simple regularizer on each latent has the biggest impact on improving performance.

We present a novel architecture for disentanglement of view-specific and shared latent subspaces in a controlled audio setting. We show that our hypothesis holds in that unique frequency information is encoded strongly into view-specific representations and common timbre information is encoded best in the shared latents. In the future we will experiment with expanding the objective function to further improve subspace-level latent disentanglement, and also plan to apply our method to real music samples.