CCStereo: Audio-Visual Contextual and Contrastive Learning for Binaural Audio Generation

Binaural audio is gaining significant attention in streaming media, revolutionising how listeners experience sound in a digital environment. This technology finds applications in various domains, including virtual reality (VR)[1], 360-degree videos[2], and music [3]. By simulating a two-dimensional soundscape, binaural audio creates a deeply immersive experience, allowing listeners to feel as though they are physically present within the auditory scene.

Binaural audio recording typically requires specialised hardware like dummy head systems [4]. These systems are costly and lack portability, making them impractical for everyday use. To address this, researchers have developed methods to spatialise audio from monaural recordings, known as binaural audio generation (BAG) [4, 5, 6]. These methods use visual information to estimate the differential audio between left and right channels. However, existing frameworks often rely on simple feature fusion strategies, which may struggle to capture complex visual-spatial relationships, limiting their generalisability and performance. To better utilise the visual information, previous works [4, 5, 6, 7, 8, 9] have explored various strategies to enhance semantic and spatial awareness across modalities. These approaches aim to improve cross-modal feature interaction [5, 6, 7, 10], strengthen spatial understanding [11, 8, 9], and incorporate 3D environmental cues [11]. Despite promising results, these methods still lack a dense spatial understanding, which limits their ability to accurately capture local spatial relationships between multimodal features.

In addition, existing models remain prone to overfitting the training environment due to their reliance on specific data distributions and insufficient regularisation mechanisms. These issues often result in limited generalisation to diverse or unseen scenarios [6]. Unfortunately, the structure of the widely used FAIR-Play [4] dataset fails to address this concern, as a significant amount of scene overlap has been observed between the training and testing sets [6], resulting in overly optimistic evaluation results on the current benchmark. Xu et al. [6] tackled this issue by reorganising the dataset based on clustering results of scene similarity. Additionally, methods involving training on synthetic stereophonic data from external sources[5, 6] and incorporating depth estimation [7] have also shown potential in mitigating the overfitting problem. However, these methods still rely on concatenation or cross-attention to control the generation process, which is less effective at integrating fine-grained conditioning information [12].

In this paper, we introduce a new U-Net-based generation framework that aims to mitigate the problems above, called Contextual and Contrastive Stereophonic Learning (CCStereo), which consists of a visually adaptive stereophonic learning method and a cost-effective inference strategy to enhance cross-modal interaction and mitigate overfitting by improving spatial reasoning. Unlike previous methods that rely solely on concatenated [4, 5, 6, 8, 13] or cross-attended [10] features for differential audio generation, we adopted the concept of conditional normalisation layers [14, 15, 12] from image synthesis field to control the generation process through estimated mean and variance shifts informed by visual context. Additionally, we propose a novel audio-visual contrastive learning method that enhances the model’s spatial sensitivity by incorporating contrastive learning across the feature maps of the anchor frame, nearby frames, and the spatially shuffled anchor frame. Moreover, the widely used sliding window inference strategy [4] introduces significant redundancy due to substantial frame overlap, which is common in video data. We argue that this overlap presents an opportunity to adopt test-time augmentation (TTA), leveraging the redundant information to enhance robustness and improve prediction accuracy. We introduce video test-time augmentation (V-TTA), which divides the video into $N$ sets of five consecutive frames and applies five-crop augmentation to each set across the entire video. To summarise, our main contributions are

• •

  A new audio-visual conditional normalisation layer that leverages visual context to dynamically align the mean and variance of the target difference audio features.

• •

  A novel audio-visual contrastive learning method that enhances spatial sensitivity by mining negative samples from randomly shuffled visual feature representations.

• •

  A cost-efficient approach for applying TTA to video data, improving robustness and prediction accuracy.

We demonstrate the effectiveness of our CCStereo model on established benchmarks, including the FAIR-Play dataset [4] with both the original 10-splits [4] and the more challenging 5-splits protocol [6]. Additionally, we extend our evaluation to the MUSIC-Stereo dataset [6], demonstrating generalisation across diverse audio-visual scenarios and superior generation quality with an efficient architecture. Code will be released.

We denote an unlabelled video dataset as $\mathcal{D}=\{(\mathbf{w}_{i},\mathbf{v}_{i})\}_{i=1}^{|\mathcal{D}|}$, where $\mathbf{v}_{i}\in\mathcal{V}\subset\mathbb{R}^{T\times H\times W\times 3}$ is a set of $T$ RGB images with resolution $H\times W$, $\mathbf{w}_{i}\in\mathcal{W}\subset\mathbb{R}^{C\times T}$ denotes the waveform data with $C\in\{L,R\}$ channels and total number of sample $T$. Given monaural audio ($\mathbf{w}^{M}_{i}=\mathbf{w}^{L}_{i}+\mathbf{w}^{R}_{i})$, We apply the short-time Fourier transform (STFT) [25] on $\mathbf{w}^{M}_{i}$, resulting in $\mathbf{a}^{M}_{i}\in\mathcal{A}\subset\mathbb{C}^{T_{s}\times F}$, where $F$ is the number of frequency bins and $T_{s}$ denotes the number of time frames. The model predicts the spectrogram of the target difference audio, defined as ($\mathbf{a}^{D}_{i}=\text{STFT}(\mathbf{w}^{L}_{i}-\mathbf{w}^{R}_{i})$).

Preliminaries. During training, we randomly sample an audio segment and its corresponding frame start at time step $t\in\mathcal{T}$ from each video (i.e., $(\mathbf{a}^{M}_{i,t},\mathbf{v}_{i,t})$) to form an input pair for the model. Our goal is to learn the parameters $\theta\in\Theta$ for the model $f_{\theta}:\mathcal{V}\times\mathcal{A}\to[-1,1]^{F\times T}$, which comprises the image and audio encoder that extract features with $\mathbf{u}^{a}_{i,t}=f_{\gamma}(\mathbf{a}^{M}_{i,t})$ and $\mathbf{u}^{v}_{i,t}=f_{\phi}(\mathbf{v}_{i,t})$, respectively, where $\gamma,\phi\in\theta$, and $\mathbf{u}^{a}_{i,t},\mathbf{u}^{v}_{i,t}\in\mathcal{U}$, with $\mathcal{U}$ denoting a unified feature space. Our approach adopted a multi-head attention block [18], which estimates the co-occurrence of audio and visual data. We simply define the cross-attention process as $\hat{\mathbf{u}}^{a}_{i,t}=f_{\text{CA}}(\mathbf{u}^{a}_{i,t},\mathbf{u}^{v}_{i,t})$, where $\mathbf{u}^{a}_{i,t}$ represent the query and $\mathbf{u}^{v}_{i,t}$ is the key and value. We decode the $\hat{\mathbf{u}}^{a}_{i,t}$ through an audio decoder $\hat{\mathbf{a}}^{D}_{i,t}=f_{\psi}(\hat{\mathbf{u}}^{a}_{i,t})\cdot\mathbf{a}^{M}_{i,t}$ , where $\psi\in\theta$. We use the MSE loss $\ell_{\text{MSE}}(\mathbf{a}^{D}_{i,t},\hat{\mathbf{a}}^{D}_{i,t})=\frac{1}{L}\sum(\mathbf{a}^{D}_{i,t}-\hat{\mathbf{a}}^{D}_{i,t})^{2}$ and magnitude loss [26] $\ell_{\text{AMP}}(\mathbf{a}^{D}_{i,t},\hat{\mathbf{a}}^{D}_{i,t})=\frac{1}{L}\sum\left\lvert\lvert\mathbf{a}^{D}_{i,t}\rvert-\lvert\hat{\mathbf{a}}^{D}_{i,t}\rvert\right\rvert$ to constrain the prediction of the difference audio, where $L=T\times F$ and $\lvert\cdot\rvert$ is the modulus of the complex number. Different from [7], we argue that even though the phase of the difference audio is inherently linked to the phases of the left and right channels, it does not fully represent the binaural phase dynamics, hence apply the phase loss [7] $\ell_{\text{PHS}}(\mathbf{a}_{i,t},\hat{\mathbf{a}}_{i,t})=\frac{1}{L}\sum\|\angle(\mathbf{a}_{i,t})-\angle(\hat{\mathbf{a}}_{i,t})\|_{2}$ on the final predicted binaural audio ($\hat{\mathbf{a}}_{i,t}$) and the ground truth binaural audio ($\mathbf{a}_{i,t}$) instead of different audio. We denote the overall reconstruction loss as $\ell_{\text{REC}}=\ell_{\text{MSE}}+\ell_{\text{MAG}}+\ell_{\text{PHS}}$.

Audio-Visual Adaptive De-normalisation (AVAD). Our AVAD adopted the concept of conditional normalisation [14, 12] that leverages the interaction between audio and visual modality to control the normalisation of decoded feature representation. The AVAD is incorporated into the U-Net decoder $f_{\psi}$ by replacing the batch normalisation layers with visually-informed normalisation layers. This modification aims to effectively integrate both spatial and semantic information into the network’s input. Please note that, for simplicity, we omit the subscripts $i$ and $t$ in the following. As depicted in Fig. 1, we first pass $\bar{\mathbf{u}}^{a}_{k}=\text{Conv}(\hat{\mathbf{u}}^{a}_{k})$ through a batch normalisation layer (BN) at $k$-th layer, and then scale and shift the normalised feature using the estimated $\alpha$ and $\beta$ via $\tilde{\mathbf{u}}^{a}_{k}=(1+\alpha)\cdot\text{BN}(\bar{\mathbf{u}}^{a}_{k})+\beta$. To compute these scaling factors, we calculate the similarity activate map $\mathbf{c}_{k}=\bar{\mathbf{u}}^{a}_{k}\cdot\text{Conv}_{v}(\mathbf{u}^{v})$ and then estimate the scale ($\alpha$) and shift ($\beta$) tensors through shared and specific MLP layers, such that $\alpha=\text{MLP}_{\alpha}(\text{MLP}_{\text{share}}(\mathbf{c}_{k}))$ and $\beta=\text{MLP}_{\beta}(\text{MLP}_{\text{share}}(\mathbf{c}_{k}))$.

Spatial-aware Contrastive Learning (SCL). The capability to learn discriminative feature presentation is crucial for the audio-visual system. One limitation of previous binaural audio generation methods is their exclusive focus on proxy tasks within the audio domain (e.g., classifying whether the audio channels are flipped), which not only under-utilises visual positional information but also hinders the learning of a joint audio-visual representation. We argue that two requirements must be satisfied to achieve effective contrastive learning: 1) spatial awareness in the learned joint representation, and 2) inclusion of a diverse set of examples. Unfortunately, previous audio-visual contrastive learning methods [20, 21] may not be suitable for the current task, as they generally failed to satisfy these two requirements. Motivated by the Jigsaw puzzle self-supervised learning method [27], we observe that these two challenges can be addressed simultaneously by introducing perturbations to the visual-spatial order.

Since the BAG problem cannot access video-level labels, we adopt a classic instance discrimination pipeline (e.g., SimCLR [28]), where each audio-visual pair is treated as its own class. For a randomly sampled minibatch of $N$ examples, we perform the contrastive prediction task on pairs of positive and negative pairs derived from the minibatch. We define the anchor set $\mathcal{E}$, positive set $\mathcal{P}$ and negative set $\mathcal{N}$ as follows:

where $p(\cdot)$ is the 2D average pooling and $S(\cdot)$ represent a shuffle process over the spatial dimension $H$ and $W$ of $\mathbf{v}_{i,t}$. Adopting the InfoNCE [28], we define the objective function as follows:

where $\mathbf{z}_{j}\in\mathcal{E}$ is an anchor feature, $\mathbf{z}_{j}^{+}\in\mathcal{P}$ is its corresponding positive pair, $\mathbf{z}_{n}^{-}\in\mathcal{N}$ are the negative features, and $\tau=0.1$ is the temperature hyperparameter.

Overall Training. The overall training objective is $\ell=\ell_{\text{REC}}+\lambda\ell_{\text{SCL}}$, where $\lambda$ is a hyperparameter.

Video Test-Time Augmentation (V-TTA). During inference, we firstly estimate the left and right complex spectrograms through $\mathbf{\hat{a}}^{L}_{i}=(\mathbf{a}^{M}_{i}+\mathbf{\hat{a}}^{D}_{i})/2$ and $\mathbf{\hat{a}}^{R}_{i}=(\mathbf{a}^{M}_{i}-\mathbf{\hat{a}}^{D}_{i})/2$. Then, we use inverse STFT (ISTFT) [25] to recover the audio signal from both channels and concatenate them together to form the final binaural waveform prediction $\mathbf{\hat{w}}_{i}=\text{Concat}[ISTFT(\mathbf{\hat{a}}^{L}_{i}),ISTFT(\mathbf{\hat{a}}^{R}_{i})]$. We use a sliding window of 0.63 seconds and a hop size of 0.1 seconds to binauralise 10-second audio clips, following an approach similar to that of the baseline methods [4]. While this process improves binaural audio generation by focusing on smaller audio segments, it introduces significant computational redundancy. Motivated by the small visual differences in 10 fps music videos, we design V-TTA to leverage this redundancy for better performance and robustness. As depicted in Fig. 2, instead of directly resizing every video frame to $448\times 224$ [4, 5, 6], we first resize each frame to $480\times 240$ and then crop a $448\times 224$ window from one of the five regions $[\text{top-left},\text{top-right},\text{bottom-left},\text{bottom-right},\text{centre}]$ based on the current frame index (i.e., “$i$ % 5”, where $i$ is a frame index). For example, if the first two audio segments are paired with the \nth5 and \nth6 frames, we crop the top-left corner of the \nth5 frame and the top-right corner of the \nth6 frame, respectively. Please refer to the Supplementary Material for additional details on sliding window integration.

We introduced CCStereo, a new audio-visual training method designed for the U-Net-based framework to enhance spatial awareness and reduce overfitting to room environments. We proposed a visually conditioned adaptive de-normalisation method that utilises the object’s spatial information to guide the decoding of the difference audio. To enhance the representation learning of spatial awareness, we design a new audio-visual contrastive learning based on mining negative samples from randomly shuffled visual feature representation. Furthermore, our cost-efficient test-time augmentation strategy enhanced robustness without adding computational overhead. Our approach consistently outperformed existing methods on the FAIR-Play and MUSIC-Stereo datasets, achieving state-of-the-art results across various metrics.