The PartialSpoof Database and Countermeasures
for the Detection of Short Fake Speech
Segments Embedded in an Utterance

In the PS scenario, we consider manipulated audio in which generated audio segments are embedded in a bona fide speech utterance and vice versa. The speaker characteristics of the embedded segments are similar to those of the true speaker. However, due to differences in the performance of TTS or VC methods, there are differences in speaker similarity. Although the PS scenario resembles other scenarios in which spoofing and CMs have already been widely studied, there are some crucial differences.

The conventional LA and DF scenarios assume that speech in a single utterance is generated using a single TTS or VC method. In the PS scenario, however, a single speech utterance may contain audio segments generated using more than one TTS or VC method, even if the entire utterance consists only of spoofed segments.

The PS scenario is also closely related to copy-moves and splicing, which are scenarios well-studied for tampering forgery [9, 10]. The copy-move method of audio forgery involves the copying and pasting of segments within the same bona fide audio sample. Splicing forgery involves assembling a speech utterance using spliced segments obtained from other audio recordings¹¹1In the TTS field, “unit selection” techniques [11, 12] apply a similar splicing operation on the basis of dynamic time warping to reduce concatenation artifacts.. Spoofed audio in the PS scenario is hence a special case of splicing forgery in which segments do not come from other bona fide audio recordings but are generated using TTS or VC.

Certain companies have started to develop technologies and services that enable users to modify specific segments of a speech recording using TTS or VC without affecting other segments, making the final utterance match as closely as possible to the original, e.g., [13, 14, 15]. Although such technologies and services are desirable for users who would like to manipulate their own speech without re-recording, they increase the possibility of misuse in the form of impersonation and fraud. They may also pose a threat to other speech-based applications such as text-dependent ASV. There is thus a need to develop new CMs that are capable of detecting partially-spoofed utterances.

During the same time as when we built the initial PartialSpoof database in 2021 [5], another database was also proposed for the PS scenario [16] for which a single multi-speaker TTS system was used to replace a single word within an utterance. This later became a part of the Audio Deep synthesis Detection (ADD) challenge database [17]. In addition to the limited number of TTS systems, the database has predetermined the target for replacement as a word, which is a form of prior knowledge even on the detection side. In contrast, we consider a spoofing scenario based on variable-length spoofed segments generated using several multi-speaker TTS and VC systems.

Since the methods used by attackers are unknown in practice, these will likely be different from the training data used to train a real system. Therefore, we should assume that TTS/VC methods for generating spoofed audio segments in the training set are mostly different from those in the evaluation set. To achieve this, each subset of the ASVspoof 2019 LA database [18] was used to construct each corresponding subset of the PartialSpoof database. It was also assumed that the attacker could use generated audio segments at a variety of acoustic units, not limited to linguistic units such as words. Hence, we used variable-length speech segments found by voice activity detection (VAD) regardless of the content of the speech.

Finally, we assumed that the attacker could carry out not advanced but basic digital signal processing (DSP) to reduce artifacts of the spoofed audio for the purpose of a reliable assessment of CMs. Since it is unrealistic to assume that advanced processing will be carried out, the DSP and VAD used by the attacker were assumed to be basic methods, and certain errors and artifacts were assumed to occur naturally.

The following is the procedure for constructing the PartialSpoof database automatically. As shown in Figure 1, the entire processing pipeline involves five stages:

The total number of samples for each temporal resolution is shown in Table II. The sample size increases as resolution increases from utterance (utt.) to 20 ms. The percentage of samples belonging to the spoof class in each temporal resolution is shown in Table III. If any part of the generated waveform is contained in the target segment to be annotated, it is labeled as spoof. We can see that the finer the temporal resolution of the segment, the less likely it is to contain generated audio signals; thus, the number of samples in the bona fide class increases at finer-grained resolutions. As can be seen from the table, the data have a bias toward the spoof class on a per utterance basis, but the extreme bias is eliminated on finer segment bases. In multi-task learning using labels at multiple temporal resolutions, such differences in bias may lead to more robust model learning.

In the PartialSpoof database, the variable-length speech segments found by the VADs are replaced without considering the meaning of sentences and words as well as the phonemes before and after the segments. Therefore, the speech in the database is not partially-spoofed speech that is intended to deceive listeners and deliver wrong linguistic messages to them but an approximation of it. On the other hand, since partially-spoofed segments of the PartialSpoof database are of variable length, CMs built on this database can execute segment-level detection at various temporal resolutions, and their accuracy can be evaluated.

A CM for the PS scenario is required to conduct utterance- and segment-level detection on an input waveform $\boldsymbol{x}_{1:T}$. The two tasks can be defined more formally as follows.

In real applications, a CM needs to assign a bona fide or spoof label to the trial by comparing the $s^{u}$ with an application-dependent threshold. It may also classify each segment on the basis of $s_{m}$, $\forall{m}\in\{1,\cdots,M\}$. In this study, we follow the conventions in the research community and do not require the CM to produce hard decisions. Given $s^{u}$ and $\boldsymbol{s}_{1:M}$ computed from test set trials, we calculate EERs to measure the utterance- and segment-level detection performance of the CM.

The definition of the utterance-level detection task in the PS scenario is identical to that in the LA (also PA and DF) scenario. Segment-level detection is different and more challenging because the required output is a sequence of scores, where each score must be derived from segments of comparatively shorter duration. Accordingly, CMs that conduct only utterance-level detection for the LA or other scenarios cannot be applied directly to the PS scenario.

Previous studies modified the utterance-level CMs for other conventional scenarios so that the CMs can do segment-level detection as well [5, 6, 16]. One approach involves the addition of a sub-network for segment-level detection to a conventional utterance-level CM. Another approach involves the application of a non-trainable re-scoring step to derive segment-level scores $\boldsymbol{s}_{1:M}$. We describe these modifications after first introducing the utterance-level CM.

One track of the ADD challenge 2022 is utterance-level detection of partially-spoofed audio [17]. Most participants used a conventional utterance-level CM, as shown in Fig. 2 (a) (e.g. [23]) whereas one team used a similar multi-task structure, as shown in Fig. 2 (b) [24].⁶⁶6Interested readers are encouraged to see the challenge website and its details at http://addchallenge.cn/add2022.

The above CMs have achieved encouraging results in existing studies [5, 16, 6], but there is still room for improvement. For example, compared with a deterministic DSP-based front-end, a trainable data-driven front-end may extract more discriminative features. As for the back-end, the commonly used LCNN only learns to conduct segment-level detection at a fixed temporal resolution. For example, using the LCNN configuration in [5], the back-end transforms the input $\boldsymbol{a}_{1:N}$ into hidden features $\boldsymbol{h}_{1:M}$, where $M=N/16$, and segment scores $s_{m}$ are computed once for every 16 frames. So we can hypothesize that it would be better to use a more flexible DNN architecture that can leverage segment labels at different temporal resolutions during training and conduct segment-level detection accordingly. The hypothesis motivated us to propose a new CM that consists of a self-supervised learning (SSL)-based front-end and a back-end that supports multiple temporal resolutions. The new front-end is expected to extract more discriminative acoustic features in a data-driven manner, while the new back-end enables the CM to better identify spoofed segments with varied length. Figure 2(e) illustrates the architecture of our CM⁷⁷7 https://github.com/nii-yamagishilab/PartialSpoof.

An SSL speech model is a DNN that processes a waveform using trainable non-linear transformations. Because they are trained using task-agnostic self-supervised criteria on speech data from various domains, pre-trained SSL models can extract robust and informative acoustic features for many down-stream tasks [25]. SSL-based front-ends have also been shown to improve CM performance for the LA scenario [25, 26, 27].

Inspired by the above studies, we use an SSL-based front-end to extract acoustic features $\boldsymbol{a}_{1:N}$ from an input $\boldsymbol{x}_{1:T}$. While there are many types of SSL models, we investigated wav2vec 2.0 [7] and HuBERT [28]. These SSL models consist of a CNN-based encoder and cascade of Transformer blocks [29]. Following previous studies [25], we extract output features from all the Transformer blocks and use their trainable weighted sum as $\boldsymbol{a}_{1:N}$. During CM training, we fine-tune the pre-trained SSL front-end in conjunction with the back-end.

Note that the number of acoustic feature frames $N$ is determined by the waveform length $T$ and CNN encoder configuration. For the pre-trained SSL models we tested, the relationship is $N=T/320$. This means that, given an input waveform with a sampling rate of 16 kHz, the extracted acoustic features $\boldsymbol{a}_{1:N}$ have a ‘frame shift’ of 20 ms.

Given $\boldsymbol{a}_{1:N}$, our back-end computes $\boldsymbol{s}_{1:M}$ at multiple temporal resolutions. This enables the proposed CM to handle the challenging segment-level detection task in a more flexible manner: by learning multiple functions $\{f_{\rho}^{\mathfrak{0}},f_{\rho}^{\mathfrak{1}},\cdots,f_{\rho}^{\mathfrak{K}}\}$, where the function $f_{\rho}^{\mathfrak{k}}$ converts $\boldsymbol{x}_{1:T}$ to segment-level scores $\boldsymbol{s}_{1:M^{\mathfrak{k}}}^{\mathfrak{k}}$ at the $\mathfrak{k}$-th temporal resolution, $\forall{\mathfrak{k}}\in\{\mathfrak{0},\cdots,\mathfrak{K}\}$. Without loss of generality, we assume there are $\mathfrak{K}$ temporal resolutions in total. Lower indices indicate finer temporal resolutions. To simplify the notation, we drop the subscript in data sequences (e.g., $\boldsymbol{s}_{1:M^{\mathfrak{k}}}^{\mathfrak{k}}$ becomes $\boldsymbol{s}^{\mathfrak{k}}$) for the rest of the paper.

As illustrated in Figure 2(e), the back-end computes $\boldsymbol{s}_{1:M}$ at different temporal resolutions in a fine-to-coarse order. First, the acoustic features $\boldsymbol{a}_{1:N}$ are used to compute the score sequence $\boldsymbol{s}^{\mathfrak{0}}\in\mathbb{R}^{N}$ at the frame level, which is the finest temporal resolution supported by the proposed CM. The $n$-th score ${s}^{\mathfrak{0}}_{n}$ indicates the likelihood that the $n$-th frame is bona fide. Next, $\boldsymbol{a}_{1:N}$ is down-sampled to $\boldsymbol{h}^{\mathfrak{1}}$ and used to compute $\boldsymbol{s}^{\mathfrak{1}}$. This process is repeated to compute segment-level scores for each temporal resolution.

For utterance-level detection, the hidden feature sequences $\boldsymbol{h}^{\mathfrak{K}}$ at the lowest temporal resolution are pooled into the utterance-level embedding vector $\boldsymbol{e}^{u}$ which is then transformed into the utterance-level score $s^{u}$. This scoring module has the same architecture as those for segment-level detection. Combined with the front-end, the proposed CM computes the scores for the PS scenario in the following order:

The scoring modules at different temporal resolutions are independent but use the same architecture, and we compare a few candidate architectures in Section VI-B. The down-sampling modules are also independent but use the same architecture. They contain a max-pooling operator with a stride equal to 2, followed by a 1D convolution layer with a kernel size of 1 and equal number of output and input channels. Therefore, if the input to the back-end $\boldsymbol{a}_{1:N}$ is of dimension $\mathbb{R}^{N\times D}$, $\boldsymbol{h}^{\mathfrak{k}}$ at the $\mathfrak{k}$-th temporal resolution will be in $\mathbb{R}^{N/2^{\mathfrak{k}}\times D}$, and the corresponding score sequence $\boldsymbol{s}^{\mathfrak{k}}$ is of dimension $\mathbb{R}^{N/2^{\mathfrak{k}}\times 1}$. This means that the temporal resolution of the $\boldsymbol{s}_{1:M}$ is reduced by a factor of 2 after each down-sampling module.

Since acoustic features are extracted every 20 ms (i.e., the ‘frame shift’ of the SSL-based front-end), a frame-level score $s^{\mathfrak{0}}_{n}$ is produced every 20 ms. This is the finest supported temporal resolution. For the $\mathfrak{k}$-th temporal resolution, one segment score $s_{m}$ is produced every $20\times 2^{\mathfrak{k}}$ ms. We set $\mathfrak{K}=5$, and the indices $\mathfrak{k}=0$ to $\mathfrak{k}=5$ correspond to temporal resolutions of 20, 40, 80, 160, 320, and 640 ms, respectively. This setting was determined empirically, taking into account the ground-truth labels provided in the extended PartialSpoof database. Although the temporal resolutions do not have explicit linguistic meanings, we hope that each time resolution may capture the following content:

• •

  40 ms: consonants and parts of vowels,

• •

  80 and 160 ms: consonants and vowels, and

• •

  320 ms: monosyllabic words and parts of longer words.

Supposing that ground-truth labels are available for all temporal resolutions, the proposed CM can be trained using two types of strategies:

All CMs were trained by minimizing the so-called P2SGrad-based mean square error [30]. This criterion was used because it was found to be more stable and slightly superior to the conventional CE used in our previous study related to the LA scenario [30]. During training, we used the Adam optimizer with a default configuration ($\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=10^{-8}$). The learning rate was initialized with $1\times 10^{-5}$ and halved every 10 epochs.

We did not use any data augmentation, voice activity detection, or feature normalization during training, nor did we trim the input trials. All experiments were repeated three times with different random seeds for CM initialization, except for the pre-trained SSL front-end. The averaged results of the three runs are reported. Each round of training was conducted using a single Nvidia Tesla A100 card.

As explained in Section V, we used a pre-trained SSL model for the front-end and designed a powerful scoring module for the back-end. In this experiment, we compared several candidates for both. Performance was measured using the utterance-level detection EER on the development set of the PartialSpoof database. The CM was trained following the same conventional utterance-level training as in the LA scenario, that is, using the single temporal resolution training strategy with utterance-level labels only.

For the front-end, we tested the four pre-trained SSL models listed in Table V; w2v2-base is based on the Wav2vec 2.0 Base model, while w2v2-large and w2v2-xlsr are based on the Wav2vec 2.0 Large model. The difference is that the Wav2vec 2.0 Base model has only 12 Transformer [29] blocks while Wav2vec 2.0 Large has 24. There are further differences in the dimension of the output features of the Transformer block, as shown in Table V. The w2v2-large and w2v2-xlsr models, both based on Wav2vec 2.0 Large, differ in the data used for pre-training. These three SSL models were included in the experiment because they have been shown to give reliable performance for the LA scenario [26]. The last candidate wavlm-large is based on HuBERT [28]. It was included because it was the top entry on the leaderboard of SUPERB⁹⁹9SUPERB: https://superbbenchmark.org/ when we started this study.

For the scoring module in the back-end, we compared the following architectures:

• •

  a single fully-connected (FC) layer;

• •

  an FC layer after a bidirectional long short-term memory (BLSTM) [39] layer;

• •

  an FC layer after two BLSTM layers;

• •

  a single gated multilayer perceptron (gMLP) block [40];

• •

  five gMLP blocks.

A gMLP block is similar to basic Multilayer Perceptrons (MLPs) with a gating unit. It was found to be simple and powerful compared with other alternatives.

We compared each of the above components in terms of the EER on the development set of the PartialSpoof database. To reduce the time cost, we used w2v2-base as the front-end when comparing the back-end scoring modules. The EERs listed in Table VI show that using five gMLP blocks gives the best performance. We then compared the SSL models while using the architecture of five gMLP blocks in the back-end scoring module. The results indicate that the three Large SSL models outperformed w2v2-base, and that w2v2-large performed best. Given these results, all following experiments were conducted using w2v2-large and five gMLP layers.

In this section, we compared the two CM training strategies, that is, training at a single temporal resolution or multiple resolutions. Performance was measured on the development set and evaluation set of the PartialSpoof database.

We first explored the two training strategies on segment-level detection. The results are shown in Figure 3(a). For the proposed CM trained at a single temporal resolution (black circle), since the segment-level ground-truth labels are available at the six segment-level temporal resolutions, we trained six versions of the proposed CM, one for each segment-level temporal resolution. After training, each CM version produces scores at the corresponding temporal resolution. We also trained the CM at multiple temporal resolutions (red triangle) that can derive scores for all resolutions at once.

Results suggest that segment-level detection is more difficult at a higher temporal resolution, but it is not impossible. For the highest temporal resolution of 20 ms, the proposed CM achieved an EER of around 10% on the evaluation set. This EER is reasonably good considering the fact that each segment consists of only one frame and is extremely short. Besides, although the multiple-resolution CM is slightly better than the single-resolution counterpart at the resolution of 640 ms, it performed worse than the single-resolution CMs at fine-grained segment levels (20 $\sim$ 320 ms). There is thus room for improvement of the fine-grained segment-level detection.

We then compared performance on utterance-level detection. The results are shown in Figure 3(b). No matter which training strategy was used, the proposed CM achieved an EER below 1% on the evaluation set for utterance level detection. Furthermore, the CM trained at multiple temporal resolutions outperformed the CMs trained at a single temporal resolution for utterance-level detection. This indicates that the coarse-grained detection tasks can benefit from the training strategies using multiple temporal resolutions.

Results of the above experiments suggest that we need to select a suitable temporal resolution and training strategy depending on our goal: (1) For the detection at the segment level, training should be applied at the specific target temporal resolution; (2) in terms of utterance-level detection, the use of more fine-grained information is expected to improve CM performance.

Besides, the differences between the EERs on the evaluation and development sets demonstrate the difficulty of segment-level detection and show that it is difficult to generalize to the evaluation set. We thus explore the differences between them in the next subsection.

We have two hypotheses to explain the significant differences between the EERs on the development and evaluation sets, and we examined both using the CM trained at multiple temporal resolutions.

(1) Hypothesis 1: More difficult spoofing systems exist in the evaluation set. For the LA scenario and ASVspoof 2019 LA database, existing studies have measured the EER over fully spoofed trials from each spoofing system and found that a voice-conversion-based spoofing system called A17 significantly increased the CMs’ EERs on the evaluation set [3, 41]. Since the PartialSpoof database used spoofed trials from the ASVspoof 2019 LA database, we hypothesize that the strong attacks in the evaluation set may have led to the higher EER. Because the segments in a partially-spoofed audio sample can be from different spoofing methods, we did not use the same investigation method as previous studies on the LA scenario and instead used a leave-one-out evaluation approach. For each spoofing method, we excluded the test trials that contain at least one segment produced by that spoofing method. We then measured the EER on the remaining trials and compared it with the original EER computed on all the test trials. We conducted the analysis on both the development and evaluation sets.

The results are shown in Figure 4. Each column in the figure corresponds to the results on segment-level detection at one temporal resolution, while each row corresponds to the results of analysis on a specific spoofing method. The IDs of the spoofing systems such as A07 were inherited from those in the ASVspoof 2019 LA database. It was observed that, if the spoofed trials that contain spoofed segments from A15¹⁰¹⁰10A15 is a hybrid spoofing system that uses a TTS voice as a source speaker and WaveNet as a vocoder for voice conversion. were removed, the EER on the rest of the evaluation set decreased significantly. For our CM, A15 was the strongest spoofing system in the PS scenario. Leaving out other spoofing systems such as A10, A11, A12 and A17 also decreased the EER but to a lesser extent. In contrast, the EERs on the development set did not decrease as much as the change in the EERs on the evaluation set no matter which spoofing method was left out.

As hypothesized, partially-spoofed segments produced by some of the spoofing systems in the evaluation set were more difficult to detect. Particularly, A15 was the strongest attack. This may be one reason that EERs in the evaluation set are higher than those in the development set. However, the EER on the evaluation set was still higher than on the development set even when we removed A15. We thus have another hypothesis to explain the EER gap between the development and evaluation sets.

(2) Hypothesis 2: Detecting spoofed segments with fewer concatenated boundaries on the evaluation set is more difficult than on the development set. As described in Section III-B, the partially-spoofed trials in PartialSpoof were created on the basis of substitution and concatenation. The overlap-add-based concatenation may bring in artifacts around the concatenated boundaries, which are expected to be useful for the CM. However, a segment to be scored by the CM may not contain any concatenated boundaries. Furthermore, if such a segment is from an unseen spoofing attack, the CM is more likely to make a mistake. Hence, we hypothesize that our CM performed worse on the evaluation set because it was not perfectly designed to distinguish the spoofed segments without any concatenation point from those with one or more concatenated boundaries. The performance on the development set may receive less impact since all the spoofing methods were seen during the training of the CM.

To verify the hypothesis, we computed the breakdown EERs on the basis of the number of concatenated boundaries in the segments. The results are listed in Figure 5. Each row in the figure lists the EERs evaluated on segments that have a corresponding number of concatenated boundaries, and each column is for EERs at one temporal resolution. We observe that the performance of our CM on the evaluation set degraded when the number of concatenated boundaries decreased. Especially, when the number of concatenated boundaries was zero, the CM’s EER significantly increased at temporal resolutions ranging from 20 to 160 ms. In contrast, the EERs on the development set varied (but to a much lesser extent) regardless of the change in the number of concatenated boundaries. As hypothesized, our CM made more errors when facing unseen spoofed segments with few concatenated boundaries in the evaluation set.

We next compared the proposed CM with the conventional CMs [16, 5, 6] explained in Section IV-B. Comparisons are made to a CM trained with the multiple temporal resolutions. EERs for the PartialSpoof evaluation set are summarized in Table VII.

The first noteworthy observation was that the conventional CMs could only directly detect segments at a single temporal resolution. This is due to the inflexible back-end structure, as discussed in Section V-A. In contrast, our proposed CM can detect spoofed segments at different temporal resolutions simultaneously. It also significantly outperformed the conventional CMs at the corresponding segment and utterance levels. These results indicate that the proposed CM is more suitable for the PS scenario.

Since a CM for the PS scenario can execute utterance-level detection, it can also be applied to the LA scenario. To determine whether the improvements brought by multi-resolution training for the PS scenario also translate to the LA scenario, we investigated the performance of the proposed CM on utterance-level detection for both LA and PS scenarios. The data for these scenarios were from the ASVspoof 2019 LA and PartialSpoof databases, respectively.

These experiments involved two training settings, one using the training set of the ASVspoof 2019 LA database and the other using that of the PartialSpoof database. For the ASVspoof 2019 LA database, since the training data only contains utterance-level labels, the proposed CM was trained using the single temporal resolution training strategy for utterance-level detection. For CMs trained using the PartialSpoof data, we used the two CMs trained using either the single temporal resolution strategy or the multiple resolution one as described in Section V-D. The first used only utterance-level labels and was trained using the same strategy as the CM trained on the ASVspoof 2019 LA database. The latter used the full set of utterance and segment labels from the PartialSpoof database and trained using the multiple temporal resolution strategy.

We used each CM to produce scores for the ASVspoof 2019 LA and PartialSpoof development and evaluation sets for the utterance-level detection. The results are shown in Table VIII. Focusing first on single resolution and utterance-level training, the performance for the ASVspoof 2019 LA database was shown to be competitive. However, the EERs increased significantly beyond 10% in the PartialSpoof dataset. This was expected because the CMs trained using the LA data are not exposed to partial spoofs, thus perform poorly. When trained using the PartialSpoof dataset; however, performance in the case of PS data improved significantly, while performance when tested using the LA data remained competitive.

Further improvements were achieved when the proposed CM was trained at multiple temporal resolutions. In summary, when trained on the PartialSpoof dataset, the proposed CM demonstrated better utterance-level detection performance for both LA and PS scenarios. Using PartialSpoof training data is beneficial even when the CM is tested on LA data.¹¹¹¹11In addition to the cross-scenario study, it is also possible to combine the two databases and see if they are complementary to each other. This investigation requires significant changes in the proposed CM structure so that model training can be effectively conducted even in situations where parts of audio files in the training database do not have segment-level labels, which is beyond the scope of this paper and is a future analysis topic.