Multi-dimensional Speech Quality Assessment in Crowdsourcing

Audio telecommunication systems, such as remote collaboration systems, smartphones, and telephones, are now ubiquitous and essential tools for work and personal use. Audio engineers and researchers have been working to improve the speech quality of these systems, with the goal of making them as good or better than face-to-face communication. However, there is still room for improvement, as it is still common to hear frequency response distortions, isolated and non-stationary distortions, loudness issues, reverberation, and background noise in audio calls.

Subjective speech quality assessment is the gold standard for evaluating speech enhancement processing and telecommunication systems, and the ITU-T has developed several recommendations for subjective speech quality assessment. ITU-T P.800 [1] describes lab-based methods for the subjective determination of speech quality, including the Absolute Category Rating (ACR). ITU-T P.808 [2] describes a crowdsourcing approach for conducting subjective evaluations of speech quality. It provides guidance on test material, experimental design, and a procedure for conducting listening tests in the crowd. The methods are complementary to laboratory-based evaluations described in P.800. An open-source implementation of P.808 is described in [3]. ITU-T P.835 [4] provides a subjective evaluation framework that gives standalone quality scores of speech (SIG) and background noise (BAK) in addition to the overall quality (OVRL). An open-source implementation of P.835 is described in [5]. Perceptual dimensions for speech quality are identified in [6] and extended to be noisiness, coloration, discontinuity and loudness [7]. Those are extensively studied in conversational test [8, 9, 10] and are focus of more recent multidimensional speech quality assessment standards namely ITU-T P.863.2 [11] and P.804 [12] (listening phase) (Table 1).

Intrusive objective speech quality assessment tools such as Perceptual Evaluation of Speech Quality (PESQ) [13] and Perceptual Objective Listening Quality Analysis (POLQA) [14] require a clean reference of speech. Non-intrusive objective speech quality assessment tools like ITU-T P.563 [15] do not require a reference, though it has low correlation to subjective quality [16]. Newer neural net-based methods, such as [16, 17, 18, 19] provide better correlations to subjective quality. NISQA [20] is an objective metric for P.804, but the correlation to subjective quality is not sufficient to use as a challenge metric.

Lab-based subjective testing in practice is slow due to the recruitment of test subjects and the limited number of test subjects, and expensive due to paying qualified test subjects and the cost of the test lab. The speed and cost result in the vast majority of research papers not using subjective tests but rather objective functions that are not well correlated to subjective opinion. An alternative to lab-based subjective tests is to crowdsource the testing. We introduce a crowdsourced multi-dimensional speech quality assessment tool that extends P.804 by adding SIG, OVRL, and reverberation (see Table 1). We show the tool is both accurate compared to lab results and is reproducible. The tool has been successfully used in the ICASSP 2023 Speech Signal Improvement challenge [21].

In Section 2 we describe the implementation of the tool. In Section 3 we provide accuracy and reproducibility analysis. In Section 4 we provide an example usage of the tool. In Section 5 we discuss conclusions and future work.

We extended the P.808 Toolkit[3] to include a test template for a multi-dimensional quality assessment. The toolkit provides scripts for preparing the test, including packing the test clips in small test packages, preparing the reliability check questions, and analyzing the results. We ask participants to rate the perceptual quality dimensions of speech namely coloration, discontinuity, noisiness, and loudness, and also reverberation, Signal Quality, and Overall quality of each audio clip. In the following, each section of the test template, as seen by participants, is described. These sections are predefined and only the audio clips under the test will be changed from one study to another.

In the first section, the participant’s eligibility and their device suitability are tested and a qualification is assigned to those that pass which remains valid for the entire experiment. The participant’s hearing ability is evaluated through digit-triplet-test [22]. Moreover, we test if their listening device supports the required bandwidths (i.e., full-band, wide-band, and narrow-band); details are in Section  2.1).

Next, the participant’s environment and device are tested using a modified-JND test [23] in which they should select which stimulus from a pair has a better quality in four questions. A temporal certificate will be issued for participants after passing this section which expires after two hours and consequently repeating this section will be required. Detailed instructions are given in the next section including introducing the rating scales and providing multiple samples for each perceptual dimension. Participants are required to listen to all samples for the first time. Figure 1 illustrates how the rating scale for quality dimensions is presented to participants. In addition, we used a Likert 5-point scale for signal quality and overall quality as specified by ITU-T Rec. P.835. In the Training section participants should first adjust the playback loudness to a comfortable level by listening to a provided sample and then rate 7 audio clips. This section is similar to the ratings section, but the platform provides live feedback based on their ratings. By completing this section a temporal certificate is assigned to the participants which is valid for one hour. Last is the Ratings section, where participants listen to ten audio clips and two gold standard and trapping questions and cast their votes on each scale. The gold standard questions are the ones that the experimenter already knows their answers (being excellent or bad) and participants are expected to vote on each scale with a minor deviation from known the answer [22]. Trapping questions are questions in which a synthetic voice is overlaid to a normal clip and asks participants to provide a specific vote to show their attention [24]. For this test, we provide scripts for creating the trapping clips, which ask participants to select answers reflecting the best or worst quality in all scales. For rating an audio clip, the participant should first listen to the end of the clip, and then they start casting their votes. During that time, the audio will be played back in a loop. After participants finish with a test set, they can continue with the next one where only the rating section will be shown until other temporal certificates are valid. By the expiration of any certificate, the corresponding section will be shown when they start the next test set.

The ICASSP 2023 Speech Signal Improvement Challenge [21] aimed to encourage research in enhancing speech signal quality in communication systems, a persistent issue in audio communication and conferencing. Participants were provided with a test and blind sets, and winners were determined through this multi-dimensional subjective test. Both the test and blind sets have 500 samples, encompassing a diverse range of speech distortions, including frequency response distortions, bandwidth limitations, reverberation, and packet loss. Overall 9 teams participated in this challenge and the reported results are based on 9x500 processed clips used in the subjective test.

We compared the correlation between quality scores collected using this survey (P.804) and P.835-based [5] subjective tests for all entries, which are reported in Table 5. A robust correlation was observed in the shared scores between the two subjective methodologies. Regarding team rankings, the only swap occurred between two teams when utilizing scores from the P.835 test, resulting in a tied rank based on P.804 ratings. Moreover, we compute the PCC between the subjective P.804 metrics and the metrics obtained using DNSMOS P.835 [18] and NISQA [20]. The correlations vary from PCC $0.478$ to $0.700$, highlighting the ongoing need for a subjective test to precisely assess speech quality.

In addition, we conducted Explanatory Factor Analysis (EFA) [25] to explore the underlying relationships among quality dimensions and assess if there’s shared variance among sub-dimensions. We applied the Maximum Likelihood extraction method with Varimax rotation, extracting three factors as suggested by the Scree plot. Bartlett’s test of sphericity yielded a significant result, and the KMO value of 0.65 indicated that the data was suitable for explanatory factor analysis. The factor loadings of quality scores on each factor are shown in Table 6. In total, three factors accounted for $62\%$ of the variance in the data. Factor 1 primarily represented signal quality, with high loadings from Signal, Coloration, and Loudness. Discontinuity formed a separate factor, with some cross-loadings from Signal, suggesting limited shared variance between Discontinuity ratings and both Coloration and Loudness. As anticipated, Noisiness constituted a distinct factor orthogonal to the others, with loading from Reverberation. Considering all mentioned factors, we highlight the importance of adding the signal and reverb dimensions to the P.804, since they contribute to orthogonal factors in a significant percentage.

Additionally, a majority of the sub-dimensions exert an influence on the quality of the signal and the overall quality. To investigate the indirect and total effects of the sub-dimensions on the overall quality, a mediation analysis was conducted, with signal quality serving as the mediator variable. The outcomes of this analysis are presented in Table 7, which reveals that Coloration had the highest total effect on the overall quality of this dataset.

This paper describes an open-source toolkit designed for multi-dimensional subjective speech quality assessment in crowdsourcing. We detail the various sections of the test template and present evidence that the collected ratings obtained using this toolkit are both valid and reproducible. Additionally, it was demonstrated that the toolkit can be used to rank speech enhancement models on large-scale subjective tests, and it can provide insights into the effect of each perceptual dimension on overall quality. Results also showed that coloration, discontinuity, and noisiness as three orthogonal factors that other dimensions load onto. Future work includes improving the survey training and scale descriptions to improve coloration and reverberation accuracy with respect to expert raters.