Voice Attribute Editing with Text Prompt

Considering the success of text-guided generation in both text and images, many recent works have explored speech generation with text prompt Guo et al. (2023); Leng et al. (2023); Ji et al. (2023). However, only a few of these works focus on specific aspects of voice characteristics Shimizu et al. (2023); Zhang et al. (2023); Yao et al. (2023), primarily addressing speech style factors such as gender, speaking speed, energy, and emotion.

In the context of methods, these studies commonly incorporated BERTDevlin et al. (2019) to extract textual embeddings from the input text prompt and utilized supervised training with reference speech to establish a mapping from textual embeddings to speaker embeddings. Prompttts2 Leng et al. (2023) introduced Diffusion Ho et al. (2020) sampling to mitigate the insufficiency of text prompt. Nevertheless, the diversity achieved during inference remains both elusive and beyond control. The proposed VoxEditor employs the ResMem and VADP blocks to address the insufficiency and imprecision issues of text prompt, allowing users to relatively modify the target voice attribute.

Several speech datasets Ji et al. (2023); Watanabe et al. (2023); Yang et al. (2023) enriched with text prompt have also been established. These datasets provided the individual text descriptions of speech style for each speech sample, which failed to convey the detailed differences in voice attributes between speakers and were unsuitable for the voice attribute editing task. In contrast, relative attribute annotations in the VCTK-RVA dataset allow the speech to be ranked across various voice attributes, facilitating alignment between independent voice attributes and the corresponding descriptors.

Memory Network Weston et al. (2015) incorporates a long-term memory module with the ability to be both read from and written to. Recently, Key-value memory has been employed for cross-modal alignment across various tasks Chen et al. (2021); Sheng et al. (2023). However, these methods often neglect the information gap between different modalities. Given the insufficiency issue of text prompt, we propose the ResMem block to bridge the gap between the text prompt and voice characteristics.

To construct a dataset suitable for the voice attribute editing task, manual annotations are necessary to express voice perception. However, systematic research on the voice perception space is currently lacking. Therefore, for practicality, we adopt a descriptor set to describe the differences in voice characteristics among speakers. Here, the descriptor set refers to the keywords commonly used in natural language to describe voice characteristics. In terms of engineering implementation, the descriptor set should be concise and cover the most commonly used voice characteristic descriptions.

Specifically, we engaged 10 speech experts with professional backgrounds in acoustic research to describe the voice characteristics of 1500 speakers based on internal 26-hour recordings in natural language. Then we merged synonyms of keywords in these descriptions and summarized the descriptor set based on word frequency statistics, as shown in Table 1.

We choose the publicly available VCTK (version 0.92) dataset Veaux et al. (2023), which has been widely utilized in VC and text-to-speech studies, as the materials for annotation. The VCTK dataset consists of speech sentences of 110 speakers, including 62 females and 48 males. Each speaker utters approximately 400 sentences in a reading style, resulting in a total of 43,475 sentences.

We hired four speech experts to conduct pairwise comparisons of voice characteristics of same-gender speakers. When presented with speech samples from $SpeakerA$ and $SpeakerB$, speech experts listened to the samples to identify the differences in voice attributes between these two speakers. Subsequently, these experts selected an unrestricted number of descriptors $\bm{v}$ from the descriptor set built in Section 3.1 to express that $SpeakerB$ exhibits more prominent voice attributes $\bm{v}$ when compared to $SpeakerA$. This forms an annotated tuple, $\{SpeakerA,SpeakerB,\bm{v}\}$, where $\bm{v}\in D\cup\text{"Similar"}$, with $D$ representing the descriptor set and "Similar" indicating that speech experts perceived the voice characteristics of two speakers as highly similar. In cases of annotation discrepancies, experts engaged in discussions to reach a final consensus. The current annotation only considers a one-way form, highlighting the more prominent voice attributes and not annotating the diminished voice attributes. Overall, through same-gender pairwise comparisons among 62 male and 48 female speakers, we collected a total of $62\times 61+48\times 47=6038$ annotated data points. In the entire dataset, the percentages of $\bm{v}$ corresponding to one, two, and three descriptors are 71.19%, 26.84%, and 1.97%, respectively. The "Similar" label accounts for 6.81 %.

We randomly selected 200 annotated samples and uploaded them on Amazon Mechanical Turk (AMT), inviting ordinary individuals to evaluate the annotations. Specifically, we provided multiple speech samples from two speakers along with our voice attribute annotations. Listeners were instructed to listen to the speech using headphones and determine whether they agreed with our annotations. A total of 40 listeners from AMT participated in the test, and their average agreement rate reached an impressive 91.78 %.

Our proposed VoxEditor adopts the end-to-end auto-encoder paradigm Li et al. (2023), that decomposes speech into content and speaker embeddings, subsequently re-synthesizing the content and edited speaker embeddings into new speech. As shown in Figure 2 (b), during inference, the LLM initially analyzes the input text prompt to obtain a specific descriptor indicating the desired voice attribute for modification. Subsequently, the descriptor embedding and speaker embedding are extracted from the descriptor and the Mel spectrograms of the source speech using a descriptor encoder and a speaker encoder. These embeddings are then input into the ResMem block (described in Section 4.2), and the output is combined through linear interpolation to derive the edited speaker embeddings. Simultaneously, the pretrained WavLM Chen et al. (2022) and prior encoder are employed to extract the content embedding from source speech. Finally, the content and speaker embeddings are fed into the decoder, which generates the edited speech. It is worth noting that when the text prompt contains multiple voice attribute descriptors, the source speech needs to be edited sequentially based on these descriptors.

During training, VoxEditor is provided with an annotated tuple $\{SpeakerA,SpeakerB,\bm{v}\}$. As shown in Figure 2 (a), it takes speech segments $Speech^{A}$ and $Speech^{B}$, along with a randomly selected voice attribute descriptor $x\in\bm{v}$, as input. These speech segments are randomly segmented from the respective speaker’s speech. In addition to the mentioned modules, the VADP block (described in Section 4.3) is employed to predict the difference degree of the specific voice attribute between two speakers. Other model modules and the autoencoder training strategy follow FreeVC Li et al. (2023), which adopts variational inference augmented with a normalizing flow and an adversarial training process. Further details about the perturb-based data augmentation, prior encoder, posterior encoder and pretraining strategy can be found in Appendix A.

Considering the insufficiency issue of text prompt, we employ a ResMem block to establish a mapping between voice attributes and their corresponding descriptors within the same feature space. Illustrated in Figure 2(c), the ResMem module accepts either the speaker embedding ${\bm{s}\in\mathbb{R}^{D}}$ or the descriptor embedding ${\bm{t}\in\mathbb{R}^{D}}$ as input, where ${\bm{s}}$ and ${\bm{t}}$ are derived from the pretrained speaker encoder and the descriptor encoder, respectively, and $D$ represents the dimension of descriptor or speech embeddings. The ResMem block is composed of a main voice-value memory $\bm{M}_{mv}\in\mathbb{R}^{M\times D}$, a residual voice-value memory $\bm{M}_{rv}\in\mathbb{R}^{N\times D}$ and a descriptor-key memory ${\bm{M}_{k}\in\mathbb{R}^{M\times D}}$, where $M$ denotes the number of the slots in ${\bm{M}_{mv}}$ and ${\bm{M}_{k}}$, $N$ denotes the number of slots in ${\bm{M}_{rv}}$ and $D$ is the dimension for each slot, which equals to the dimension of the descriptor and speaker embeddings.

${\bm{M}_{mv}}$ is designed to capture the main voice characteristics that can be articulated with voice descriptors, while ${\bm{M}_{rv}}$ is utilized to address aspects of voice characteristics that are challenging to describe in text. Specifically, when a speaker embedding ${\bm{s}}$ is given as a query, the cosine similarity between the query and each slot in ${\bm{M}_{mv}}$ is computed, followed by softmax normalization function, expressed as follows,

$$w_{mv}^{i}=softmax(\frac{\bm{s}^{\top}\bm{m}_{mv}^{i}}{\left\|\bm{s}\right\|_{2}\left\|\bm{m}_{mv}^{i}\right\|_{2}}),$$

(1)

where $\bm{m}_{mv}^{i}$ denotes the $i$-th slot in ${\bm{M}_{mv}}$ and $w_{mv}^{i}$ represents the degree of relevance between the $\bm{m}_{mv}^{i}$ and the speaker embedding ${\bm{s}}$. We then obtain the cosine similarity vector ${\bm{w}_{mv}}=[w_{mv}^{1},w_{mv}^{2},\cdots,w_{mv}^{M}]^{\top}\in\mathbb{R}^{M}$ by computing cosine similarity with all slots. Next, we generate the recalled main speaker embedding as follows,

$$\hat{\bm{s}}_{m}=\bm{M}_{mv}^{\top}\bm{w}_{mv}.$$

(2)

Similarly, we generate the recalled residual speaker embedding as follows,

$$w_{rv}^{j}=softmax(\frac{\bm{s}^{\top}\bm{m}_{rv}^{j}}{\left\|\bm{s}\right\|_{2}\left\|\bm{m}_{rv}^{j}\right\|_{2}}),$$

(3)

$$\bm{w}_{rv}=[w_{rv}^{1},w_{rv}^{2},\cdots,w_{rv}^{N}]^{\top}\in\mathbb{R}^{N},$$

(4)

$$\hat{\bm{s}}_{r}=\bm{M}_{rv}^{\top}\bm{w}_{rv},$$

(5)

where $\bm{m}_{rv}^{j}$ denotes the $j$-th slot in ${\bm{M}_{rv}}$. Then, we obtain the recalled speaker embedding $\hat{\bm{s}}$ and calculate the mean square error (MSE) between $\bm{s}$ and $\hat{\bm{s}}$ as well as the MSE between $\bm{s}$ and $\hat{\bm{s}}_{m}$.

$$\hat{\bm{s}}=\hat{\bm{s}}_{m}+\hat{\bm{s}}_{r},$$

(6)

$$\mathcal{L}_{rec}=\|\bm{s}-\hat{\bm{s}}\|_{2}^{2}+\|\bm{s}-\hat{\bm{s}}_{m}\|_{2}^{2}.$$

(7)

In this manner, the slots within the ${\bm{M}_{mv}}$ and ${\bm{M}_{rv}}$ can serve as foundational vectors for constructing the entire voice characteristics space, allowing for various combinations of these slots to represent a wide range of voices.

Then, we utilize the slots in $\bm{M}_{mv}$ as a streamlined bridge to map descriptor embeddings onto the voice space. In detail, given the descriptor embedding $\bm{t}$, we generate the recalled descriptor embedding $\hat{\bm{t}}$ in a similar way as the recalled main speaker embedding as follows,

$$w_{t}^{i}=softmax(\frac{\bm{t}^{\top}\bm{m}_{k}^{i}}{\left\|\bm{t}\right\|_{2}\left\|\bm{m}_{k}^{i}\right\|_{2}}),$$

(8)

$$\bm{w}_{t}=[w_{t}^{1},w_{t}^{2},\cdots,w_{t}^{M}]^{\top}\in\mathbb{R}^{M},$$

(9)

$$\hat{\bm{t}}=\bm{M}_{mv}^{\top}\bm{w}_{t},$$

(10)

where $m_{k}^{i}$ denotes the $i$-th slot in $\bm{M}_{k}$, cosine similarity is calculated with the slots in the descriptor-key memory ${\bm{M}_{k}}$ and aligned with the main voice-value memory ${\bm{M}_{mv}}$. In this way, descriptor embeddings are mapped to the main voice characteristics space, with the slots in $\bm{M}_{mv}$ serving as the basis vectors.

Considering the imprecision inherent in text prompt, we propose the VADP block, designed to predict the difference degree of the specific voice attribute between two speakers. Voice characteristics can exhibit local variations Zhou et al. (2022) due to factors such as content, rhythm, and emotion. Therefore, we assume that the differences in voice attributes between different speech samples from two speakers are not deterministic but follows a Gaussian distribution.

As shown in Figure 2(a), given the speaker embedding ${\bm{s}^{A}}$ from $Speech^{A}$, speaker embedding ${\bm{s}^{B}}$ from $Speech^{B}$ and descriptor embedding $t$, we concatenate three embeddings to obtain a cross-modal embedding, We then use linear layers and ReLU activation functions to predict the mean and variance of a Gaussian distribution. Subsequently, we sample from this Gaussian distribution to obtain the difference degree of specific voice attributes, which we map through a sigmoid activation function to derive the editing degree $\alpha\in[0,1]$. Similar to Imagic Kawar et al. (2023), we linearly interpolate between the recalled descriptor embedding $\hat{\bm{t}}$ and the recalled main speaker embedding $\hat{\bm{s}}_{m}^{A}$ from $Speech^{A}$ to derive the edited speaker embedding ${\bm{s}_{edit}}$,

$${\bm{s}_{edit}}=\alpha\cdot\hat{\bm{t}}+(1-\alpha)\cdot\hat{\bm{s}}_{m}^{A}.$$

(11)

Additionally, we align the slot-weights distributions between the recalled main speaker embedding $\hat{\bm{s}}_{m}^{B}$ and ${\bm{s}_{edit}}$ using Kullback-Leibler (KL) divergence,

$$\mathcal{L}_{align}=D_{KL}(\bm{w}_{mv}^{B}||\alpha\cdot\bm{w}_{t}+(1-\alpha)\cdot\bm{w}_{mv}^{A}),$$

(12)

where $\bm{w}_{mv}^{A}$ and $\bm{w}_{mv}^{B}$ denote the cosine similarity vectors in the main voice-value memory ${\bm{M}_{mv}}$ for $Speech^{A}$ and $Speech^{B}$, respectively. In this way, we explicitly align the voice attributes with their corresponding descriptors.

As shown in Figure. 2(b), we utilize a LLM (GPT-3.5-TURBO) to scrutinize the input text prompt and extract target voice attribute descriptors. In cases where the descriptor is absent from the set, the LLM is employed to locate the closest matching descriptor within the set for substitution. For further elucidation, please refer to Appendix B. Next, we input both the target voice attribute descriptor and source speech to obtain the recalled descriptor embedding $\hat{\bm{t}}$, recalled main speaker embedding $\hat{\bm{s}}_{m}$ and recalled residual speaker embedding $\hat{\bm{s}}_{r}$. Subsequently, we achieve edited speaker embedding through linear interpolation,

$${\bm{s}_{edit}}=\alpha\cdot\hat{\bm{t}}+(1-\alpha)\cdot\hat{\bm{s}}_{m}+\hat{\bm{s}}_{r},$$

(13)

where the value of editing degree $\alpha$ is initially set to its recommended value 0.7 (refer to Figure 5). The editing degree can be further adjusted within the range $[0,1]$, and increasing $\alpha$ will progressively enhance the prominence of the target voice attribute.

Our experiments were conducted using the VCTK-RVA dataset, consisting of 98 speakers for both the training and validation sets. Among these, 200 sentences were randomly selected for validation, while the remaining sentences were utilized for training. For testing, speech samples were categorized into two sets: the seen speaker set, comprising speakers from the training set, and the unseen speaker set, consisting of speakers not encountered during training. Each set comprised 12 speakers, each contributing three sentences. During the evaluation, source speech underwent voice attribute editing for each voice attribute in the descriptor set, with $\alpha$ varying from 0 to 1 in increments of 0.1. We devised 10 predefined sentence patterns, such as "I want this sound to become more [Descriptor]". For each edit, a sentence pattern was randomly chosen, and [Descriptor] was replaced with the target voice attribute descriptor to form the text prompt.

All speech samples were downsampled to 16k Hz. Linear spectrograms and 80-band Mel spectrograms were computed using a short-time Fourier transform (STFT) with FFT, window, and hop sizes set to 1280, 1280, and 320, respectively. The dimensions of descriptor embeddings, speaker embeddings and slots were equal to $D=256$. The numbers of slots in the ResMem Block were set to $M=32$ and $N=4$. We put more information about $M$ and $N$ in Appendix C.

As pioneers in addressing the voice attribute editing task with no existing comparable methods, we conducted a thorough comparative analysis of our proposed method against the following baseline and ablation models to evaluate its effectiveness: (1) PromptStyle Liu et al. (2023): We utilized its style embedding generation module to replace the edited speaker embedding generation module in VoxEditor. Specifically, the original prompt encoder was modified to a speaker encoder and a descriptor encoder. MSE loss was employed to minimize the distance between $\bm{s}^{A}+\bm{t}$ and $\bm{s}^{B}$. (2) w/o Voice Res., (3) w/o ResMem, (4)w/o VADP. More details about these ablation models can be found in Appendix D. All objective and subjective evaluation results are summarized in Table. 2.

We can observe that VoxEditor outperformed other methods significantly in all metrics (${p<0.05}$ in paired t-tests) except for MOS-Nat. When the ResMem block was not utilized (PromptStyle and w/o ResMem), the model’s performance sharply declined. This is attributed to the inability to effectively align the voice attributes with their corresponding descriptors. Furthermore, the TVAS metric for same-gender speakers with different editing degrees $\alpha$ is illustrated in Figure 3. We noticed that as $\alpha$ increases, the speech generated by our method became increasingly prominent in the specified voice attributes. In contrast, for the PromptStyle method, the direction of editing was not consistent with the specified voice attributes.

We found that the performance of the w/o Voice Res. was comparable to that of our method in terms of TVAS and MOS-Cons, but there was a significant difference in MOS-Corr. For same-gender speakers, the voice characteristics of the generated speech by w/o Voice Res. were very similar when edited in the same voice attribute with $\alpha$ approaching 1, and the correlation in voice characteristics between the generated speech and the source speech was almost nonexistent. The removal of the VADP block prevented the model from modelling refined edited speaker embeddings during training, resulting in compromised descriptor embeddings. Consequently, during inference, even when the editing degree approached 1, the edited voice attributes were not prominent enough, leading to a decrease in the TVAS and MOS-Cons scores. In addition, since these methods follow the same auto-encoder paradigm, their performance in terms of MOS-Nat was quite comparable.

We randomly selected an additional set of 100 utterances from an unseen speaker and visualized the speaker embeddings of the generated speech edited with different voice attributes through t-SNE Chan et al. (2019), as shown in Figure 4. We observed that, as the editing degree $\alpha$ increased, generated speech edited with the same target voice attribute gradually formed distinct clusters, which demonstrated the stability of our method. Additionally, due to variations in the number of voice attribute annotations and differences in the prominence of voice attributes, there were variations in the editing performance of different voice attributes. When $\alpha$ equalled 0.3, the speech edited on the "Hoarse" attribute already exhibited clear voice features, forming clusters, while generated speech edited with other voice attributes predominantly retained the voice characteristics of the source speech.

While the optimal value for editing degree $\alpha$ may vary depending on different requirements, we aim to determine the optimal editing range for $\alpha$. The ideal voice attribute editing should involve a change toward the specified voice attribute direction while still preserving some voice characteristics of the source speaker. To this end, we selected an additional 100 speech samples from unseen speakers and performed editing with weights $\alpha$ ranging from 0 to 1 across various attributes. We evaluated the MOS-Cons and Mos-Corr scores of all generated speech and show the average results in Figure 5.

We observed that when the editing degree was less than 0.4, the generated speech closely resembled the source speech, and the editing had a small impact. For $\alpha\in[0.6,0.8]$, the generated speech aligned well with the text prompt while also preserving the some voice characteristics of source speech. However, when the weight exceeded 0.8, there was a significant decrease in the similarity of voice characteristics between the generated speech and the source speech. Therefore, we considered the optimal editing range to be between 0.6 and 0.8, setting the recommended value of $\alpha$ to 0.7.