BANC: Towards Efficient Binaural Audio Neural Codec for Overlapping Speech

Neural audio codecs (NACs) are used to compress audio signals into codes to reduce the amount of data transmitted or stored. Existing NACs can be categorized into hybrid approaches and end-to-end NACs. Hybrid NACs combine traditional signal processing-based audio coding methods with neural speech synthesis architectures [1, 2, 3]. Recently, data-driven end-to-end NAC architectures have been proposed [4, 5, 6, 7], substantially improving the quality of reconstructed audio. The end-to-end approach does not assume the nature of the input audio training dataset, allowing it to generalize to any audio content. Most NACs are composed of encoder, quantizer, and decoder modules. Current NACs are designed to compress single-channel or stereo audio signals, and their decoder modules are typically optimized for single-speaker scenarios [7]. Two major drawbacks are evident in prior NAC designs: 1) They predominantly focus on single-channel audio processing. 2) They assume the presence of only one primary speaker within the audio. To address these limitations, our paper introduces a novel neural spatial audio codec model, aptly named BANC. This model efficiently compresses binaural speech, accommodating both single-speaker and two-speaker scenarios with different spatial locations.

A key distinction between single-channel and binaural audio is the latter’s encapsulation of spatial localization, in addition to clean speech content ($S_{C}[t]$) [8]. This spatial context manifests in various acoustic facets, such as early reflections, late reverberations, diffractions, as well as interaural discrepancies like interaural time difference (ITD) and interaural level difference (ILD) between microphones. Mathematically, these intricacies can be captured by the impulse response (IR) function. Consequently, for a mono-speaker scenario, one can decompose the clean speech content $S_{C}[t]$ and the binaural acoustic effect $BIR[t]$ from the binaural speech $S_{B}[t]$ as:

For overlapped binaural speech ($S_{OB}[t]$) with two speakers in two different spatial locations, we can separately decompose their clean speech contents ($S_{C1}[t]$, $S_{C2}[t]$) and their binaural IRs (BIRs) ($BIR_{1}[t]$, $BIR_{2}[t]$) as follows:

Main Contribution: We present a pioneering NAC architecture optimized for binaural overlapped speech from two speakers, crucially retaining each speaker’s spatial context. This architecture is illustrated in Fig. 1. In contrast to the existing AudioDec model[7], our key contributions are: 1) Expansion of previous neural codecs from single to binaural audios. 2) Enhanced capability for compressing and decoding overlapping speech. 3) A novel strategy of separately compressing speech content and spatial cues, ensuring post-decoding preservation of spatial context for each speaker. 4) Achieving a high compression rate for binaural audio. Specifically, our model can reconstruct a 48 kHz binaural speech signal from two distinct speakers using only 12.6 kbps bandwidth, a feat surpassing AudioDec which operates at 24 kbps for binaural speech and results in a remarkable 52% reduction in spatial errors, respectively. Moreover, BANC demonstrates superiority over both Opus[9] and Encodec [5] in quantitative and qualitative analysis when tested under comparable bandwidths. We provide reconstructed speech samples, spectrograms and source code for future research at https://anton-jeran.github.io/MAD/.

Traditional audio codecs: Linear predictive coding-based audio codecs [10, 11] and model-based audio codecs [12] have been proposed in the past for speech coding, but their quality is limited. Among traditional methods, Opus [9] and EVS [13] are state-of-the-art traditional audio codec architectures, and they can support different bitrates and sampling rates at high coding efficiency in real-time.

Neural audio codecs (NAC): End-to-end data-driven architectures have been proposed to encode mono and stereo audio with impressive performance [4, 5, 6]. Encodec [5] compresses stereo audio by processing the left and right channels separately, which leads to inefficient compression as the same speech content is encoded twice in both channels. Our BANC model reduces bandwidth by encoding speech content only once and is optimized to efficiently compress overlapping speech while preserving each speaker’s speech content and spatial acoustic features.

Speech dereverbation and RIR Estimation: Recently, NAC-based architectures have been proposed for audio-visual speech enhancement [14]. Similarly, in our work, we decode clean speech from binaural speech. Generative architectures have also been proposed to estimate IR from given spatial information [15, 16]. Encoder-decoder architectures have shown promising results in estimating IR from reverberant speech [17, 18]. We propose a neural codec to estimate the IR of a one-second duration from binaural speech.

We propose BANC to compress binaural speech $S_{B}(x)$ with a sampling rate of 48 kHz. Similar to typical NAC [4, 7], our model consists of an encoder, projector, quantizer and decoder modules. We propose simple and complex decoder architecture for single-speaker and two-speaker scenarios, respectively. Our proposed encoder architecture is the same for single-speaker and two-speaker cases. We adapt the projector and quantizer from the AudioDec [7].

Dataset: We generate binaural speech datasets for both single-speaker and two-speaker scenarios using clean speech from the Valentine dataset [22, 23] and simulated BIRs from pygsound [24]. We simulated 50k BIRs using pygsound, which were then randomly convolved with the clean speech corpus using Eq.1 for single-speaker and Eq.2 for two-speaker scenarios. We split the simulated dataset into 33,975 training samples, 750 validation samples, and 752 test samples.

We measure our model’s ability to preserve interaural time difference (ITD) and interaural level difference (ILD) in reconstructed binaural speech. As proposed in prior work [28], we use the generalized cross-correlation phase transform (GCC-PHAT) algorithm [29] to calculate the ITD error (Eq. 10) between the left and right channels of the ground truth speech ($B^{L}$, $B^{R}$) and the reconstructed speech ($\hat{B}^{L}$, $\hat{B}^{R}$).

We define the ILD error for left channel ($\mathbf{E_{ILDL}}$) and right channel ($\mathbf{E_{ILDR}}$) as follows:

Table III shows the BIR estimation error for our approach. We observed that improving the binaural speech estimation quality using the HiFi-GAN vocoder and spectrogram loss indirectly contributed to better BIR estimation, reducing the overall error for $T_{60}$, DRR, EDT, and CTE by 6.9%, 62.1%, 56.6%, and 64%, respectively, in the single-speaker scenario. The performance of our two-speaker model is comparable to that of our single-speaker model BANC-V1.

We propose BANC, a novel binaural NAC for single-speaker speech and two-speaker spatially overlapped speech. Our approach outperforms traditional methods and NACs with similar bandwidth, preserving binaural acoustic effects by up to 52%. We introduce a novel technique to compress speech content and acoustic effects separately, demonstrating that our method can reduce the bandwidth for compressing binaural speech by 48% compared to compressing each channel individually using AudioDec. Given the complexity of the scenario and the need for headphone-based evaluation, we have tested our approach on binaural two-speaker spatially overlapped speech. In the future, we aim to extend it to compress and decode overlapped speech from multiple speakers in arbitrary locations.