FreeCodec: A disentangled neural speech codec with fewer tokens

As illustrated in Fig.1(a), our proposed method consists of three components: encoders, quantizers, and decoders. Unlike existing works, our encoder proposes a more detailed modeling focus on different intrinsic properties in human speech. It can alleviate the pressure on a single encoder component and provides a more detailed capability to model intrinsic speech attributes. Specifically, we introduce three types of encoders to encode content, prosody, and timbre information, respectively. The input raw speech signal is first used to generate three latent feature representations. Then the quantization layers produce compressed representations. Finally, the decoders reconstruct the speech signal from the compressed latent representations. In addition, we utilize different training strategies to provide three versions for reconstruction(zero-shot TTS, speech coding) and disentanglement(voice conversion) scenarios. The details of training strategies are described in Section II-C and II-D.

Speaker Encoder Existing approaches extract a global embedding in the encoder in an unsupervised manner, and then the decoder aggregates the global embedding into the frame-level embeddings. They assume that the global embedding can represent time-invariant information, such as the characteristics of the speaker and speaking style. Here, we follow this unsupervised manner and further extract the speaker’s information more precisely. We utilize a pre-trained speaker encoder, ECAPA-TDNN [20], a state-of-the-art speaker recognition network based on convolution neural networks with an attentive statistics pooling layer. The mel-spectrogram sampled from the raw speech signal is fed into the speaker encoder to get one global timbre vector.

Content Encoder The architecture of the content encoder follows the SuperCodec [6] encoder using (2, 4, 5, 8) as strides, a number ${B}_{enc}$ = 4 of convolution blocks. It indicates a total downsampling of 320 times and outputs 256-dimensional content features with a frame rate of 50 Hz from 16 kHz speech. In order to reduce the redundancy of the content encoder, we propose to use a self-supervised model to explicitly model the content information, as shown in Fig.1(a).

Prosody Encoder The prosody encoder extracts the information apart from the speaker and content information, as shown in Fig.1(b). In [21, 22], the first 20 bins in each mel-spectrogram frame are taken as input to extract prosody because it contains almost complete prosody and much less speaker and content information than the full band. Following the related work [22], we adopt the prosody encoder consisting of two convolution stacks, a max pooling layer with a stride of 8 to remove content and speaker information further. Our proposed method sets the FFT and hop size to 1024 and 320. With these setups, the prosody encoder results in roughly a frame rate of 7 Hz feature embeddings with 256 dimensions.

We adopt different methods to quantize different features. For the content and prosody information, we adopt a plain vector quantizer with one codebook, and the codebook size is set to 256. As for the speaker embedding, we use two types: continuous representation for FreeCodec-v1 and FreeCodec-v3 and discrete representation for FreeCodec-v2. Specifically, we compress the speaker embedding by group vector quantization (GVQ) in FreeCodec-v2 for speech coding. It divides the speaker embedding into eight groups that are quantized by one codebook with 1024 codebook size respectively. As for FreeCodec-v1 and FreeCodec-v3, we provide the continuous representation to the decoder for better reconstruction in such as zero-shot TTS and voice conversion scenarios, similar to [23, 24].

FreeCodec also employs a mirrored decoder upsampling structure²²2https://github.com/exercise-book-yq/Supercodec. It uses (8, 5, 4, 2) as strides, resulting in a total upsampling of 320 times. However, reconstructing in the high compression rate scenarios is particularly challenging. Before upsampling, we first adopt 4 layers Transformer encoder to enhance the semantic modeling. Then, we use ConvNeXt [26] as a fundamental backbone to condition the prosody and speaker representations.

We incorporate adversarial training to promote perceptual quality, using a multi-scale STFT-based (MS-STFT) discriminator. The training loss of the proposed method comprises five components: reconstruction loss, VQ commitment loss, content loss, feature matching loss, and adversarial losses. The reconstruction loss, feature loss, and adversarial losses follow EnCodec [3].

For the content loss, we extract the last layer representation from a pre-trained WavLM-Large model [27] as the semantic learning target at 50 Hz. In FreeCodec-v1 and FreeCodec-v2, we use it to reduce the redundancy of the content encoder. It maximizes the cosine similarity at the level of the dimensions across all timesteps between the outputs of the content encoder and semantic learning target. In FreeCodec-v3, we only use the semantic learning target at the decoder to prevent additional speaker information from leaking to the content encoder and quantizer. We also utilize spectrogram-resize based data augmentation on the prosody and content encoder in the training [28]. This approach achieves better performance in disentanglement scenarios.

We trained our model on LibriSpeech [29], which consists of approximately 1000 hours of speech at 16khz. We use train-clean-100, train-clean-360, and train-other-500 subsets. For a fair comparison, we adopt two recent neural codecs, TiCodec³³3https://github.com/y-ren16/TiCodec and Descript-audio-codec (DAC)⁴⁴4https://github.com/descriptinc/descript-audio-codec, which have demonstrated success in the domain of neural speech codecs. The baselines are re-trained with 1 and 2 codebooks, indicating 0.5 kbps and 1 kbps. FreeCodec and TiCodec are trained on two V100 GPUs with 400 k iterations and batchsize of 20 per GPU. DAC is trained on two V100 GPUs with 800 k iterations and batchsize of 10 per GPU. In addition, we also consider several open-source speech codecs as baselines, EnCodec ⁵⁵5https://github.com/facebookresearch/encodec at 3 kbps, and Lyra-v2 ⁶⁶6https://github.com/google/lyra at 3.2 kbps, and Speechtokenizer[16] at 3 kbps, and SemantiCodec[18] at 1.3 kbps, and Wavtokenizer-small[8] at 0.9 kbps. For Encodec and Wavtokenizer-small, we use the 24 kHz pre-trained model to synthesize speech, corresponding to the same compression rate of 2 kbps and 0.6 kbps for the 16 kHz sampling rate, respectively.

As for the voice conversion, three baseline models are selected to be compared with FreeCodec-v3: VQMIVC [31], YourTTS [32], Wav2vec-vc [33]. These models are trained on the VCTK datasets.

We evaluate FreeCodec from two aspects: 1) Reconstruction Quality. We conduct it on VCTK [30] and test-clean subset of LibriSpeech. For VCTK, we randomly select data from 8 speakers and 2911 utterances for the test. For LibriSpeech, we use the test-clean subset, 2620 utterances for the test. All audio samples are downsampled to 16 kHz. 2) Disentanglement Ability. we evaluate it based on the voice conversion benchmark. We randomly select 200 utterances from LibriSpeech test-clean subset as source speech and 6 speakers from VCTK as the target speaker. All models are evaluated in LibriSpeech Test-clean to VCTK scenarios.

Table I summarizes the results of objective reconstruction experiments. The FreeCodec-v1 performs best in almost all objective metrics in test sets. Especially in out-of-domain environments, our proposed method achieves the best reconstruction performance using only approximately 57 tokens per second. Compared to the FreeCodec-v2, the FreeCodec-v1 is better especially in speaker similarity. It shows that the continuous global representation is more effective in reconstruction scenarios. Although slightly lower STOI and SECS than DAC at 1 kbps, FreeCodec-v2 gets better objective speech quality according to UTMOS and WARP-Q. The same result can also be concluded in subjective evaluation, as illustrated in Fig. 2. In addition, FreeCodec-v2 achieves higher quality than Lyra-v2 at 3.2 kbps, EnCodec at 3 kbps, and TiCodec at 1 kbps. It demonstrates that the more-grained disentanglement framework leads to lower bitrate but higher reconstruction quality.

Furthermore, we also conducted an ablation study to validate the explicit effect of content loss in the content encoder. It can be observed that removing the content loss causes the performance drop in all objective metrics, especially the UTMOS and STOI.

In this section, we describe the disentanglement ability on the voice conversion experiments. FreeCodec-v3 achieves voice conversion by using the speaker information from the target speech. As shown in Table II, the FreeCodec-v3 achieves lower WER and CER than all baseline models, especially the text-based models. Meanwhile, the F0 PCC and speaker similarity of FreeCodec-v3 is also the highest. This indicates that our proposed method achieves superior disentanglement in intrinsic attributes of human speech.