EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge

Traditional TTS evaluation rely on humans to provide a Mean Opinion Score (MOS) that is both costly and statistically noisy, due to its reliance on a changing pool of evaluators. Recent advances in automatic TTS evaluation typically rely on two metrics: the Word Error Rate (WER) seed-tts ; xu2025qwen25omnitechnicalreport ; lajszczak2024base ; cosyvoice , as calculated by using an ASR model to convert the generated speech back into text and compare with the reference text; a speaker-similarity score (SIM) seed-tts ; xu2025qwen25omnitechnicalreport ; lajszczak2024base ; sparktts , calculated by comparing the latent embeddings of generated vs. reference speech using an audio foundation model, such as WaveLM wavelm . Recent works also explored the use of models to directly predict MOS (Sim-MOS) by training on datasets such as The Samsung Open MOS Dataset Maniati_2022 and The VoiceMOS Challenge voicemoschallenge .

While these metrics serve to capture how natural or accurate a system sounds, their evaluation power is limited by the difficulty and expressivity of the voice dataset and cannot handle nuanced, context-sensitive phenomena such as emotional prosody or complex syntax. More recently, BASE-TTS lajszczak2024base introduced an emergent abilities test suite that probes seven linguistically motivated phenomena, such as compound nouns, emotions, foreign words, paralinguistics, etc., using 20 hand-crafted prompts per category. Although BASE-TTS shifts the focus toward higher-order TTS capabilities, its dataset size is limited and reliance on human expert judgers is costly.

Our work addresses these limitations by automating test-generation and expanding category coverage. In particular, we create progressively harder stimuli at scale to differentiate between high-performing TTS systems. Our framework thus bridges the gap between traditional metric-based evaluation and nuanced, reproducible benchmarking.

A key weakness in previous benchmarks is the need for human judges. Recent years have seen a growing trend of integrating audio encoders with LLMs. This has resulted in large audio-language models (LALM) that excel at a variety of audio comprehension tasks wavelm ; geminiteam2025geminifamilyhighlycapable ; rubenstein2023audiopalmlargelanguagemodel ; tang2024salmonngenerichearingabilities ; chu2023qwenaudioadvancinguniversalaudio . SALMONN tang2024salmonngenerichearingabilities uses finetuned LALM to predict MOS, SIM and A/B testing scores. Wang et al. wang2025enablingauditorylargelanguage extends this by finetuning an LALM to also generate open-ended qualitative feedback, covering noisiness, distortion, prosody, etc., alongside scores. This approach leverages the LLM’s contextual knowledge to provide multi-dimensional evaluations more akin to a human expert. Chen et al. chen2025audio compiled the first corpus of human-written TTS evaluations (with overall MOS plus detailed error annotations) and used it to train an audio-augmented GPT model. The resulting system can describe speech quality degradations and compare two samples in free-form language and outperforms prior state-of-the-art MOS prediction models. WavReward ji2025wavrewardspokendialoguemodels employs a generalist reward model to score spoken dialogue quality across dimensions like clarity and expressiveness .

Our work not only use LALMs as judges but also to generate tests spanning categories of emergent TTS abilities. Our evaluation demonstrates that even out-of-the-box LALMs like Gemini-2.5 Pro are capable of evaluating emergent capabilities in SOTA TTS systems and produce A/B testing results that are highly-correlated with human preference.

We follow two key guidelines when constructing text prompts in EmergentTTS-Eval: (1) the dataset should encompass real-world challenges faced by TTS systems, and (2) it should exhibit varying levels of difficulty to enable fine-grained assessment of system capabilities. To this end, we begin with a diverse set of seed prompts and iteratively expand their scope (breadth) and complexity (depth). This reflects the approach of progressively increasing instruction difficulty used in instruction tuning xu2023wizardlm . Our seed prompts are derived from a collection of 140 human curated samples introduced in the BASE-TTS (lajszczak2024base, ). These samples span 7 challenging TTS categories commonly encountered in real-world scenarios: “Compound Nouns”, “Emotions”, “Foreign Words”, “Paralinguistics”, “Questions”, “Syntactic Complexities”, and “Punctuation”, with 20 samples per category. Some prompts pose challenges for TTS systems because generating realistic speech requires a deep understanding of the text’s semantic context. Consider the following text prompt from the “Emotions” category: A profound sense of realization washed over Beal as he whispered, "You’ve been there for me all along, haven’t you? I never truly appreciated you until now.". An effective TTS system should recognize the emotional context and appropriately render the quoted sentence as a whisper to reflect Beal’s sentiment.

Although the BASE-TTS proposed set is of high quality, it is limited in its ability to explore the depth within each category and lacks broad diversity, as it was curated by a small group of individuals. However, complexity and diversity are essential for a robust evaluation benchmark, as they help assess challenging scenarios where system failures can significantly impact user experience. For example, we want to evaluate on real-world scenarios like sequential interrogative questions, sustained emotion synthesis with natural shift to contrasting emotions, multi-code switching, etc. In addition to the categories defined in BASE-TTS, we introduce a new category called “Complex Pronunciation”, which contains prompts featuring unusual characters, numerals, and tongue-twisters. We exclude the “Compound-Nouns” category due to it’s limited scope and the strong performance of current TTS systems according to manual assessment. We also drop the “Punctuations” category, as punctuation-related challenges are inherently addressed within other categories such as “Paralinguistics”, “Syntactic Complexity”, and “Complex Pronunciation”.

To enrich the complexity of the text prompts and improving their diversity, we leverage LLMs to iteratively refine the initial utterances. The LLM is first tasked to curate samples that improve the dataset breadth-wise, guided by explicit diversity enhancement criteria embedded in the prompt and reinforced by strict structural constraints. Afterwards, we apply an iterative refinement process to construct a multi-tiered dataset encompassing utterances of varying complexities. In the process, we take the base utterance $U_{i}$, and create a deeper version $U_{i+1}$ through a specific refinement method for each category. $U_{i+1}$ can then be fed as input to the next refinement step, to get an even more challenging $U_{i+2}$ and so on. According to our experiments, the LLMs are able to generate strong refinements if we provide detailed criteria in the instruction and three refinement steps are sufficient. We share the prompts we use for all the categories in the Appendix A, and an example refinement for Paralinguistics and Foreign Words category is shown in Figures 2 and 2, respectively. Here are the description of the six categories in the final dataset: {description*}

Contain sequential questions and statements. This evaluates the TTS system’s ability in generating interrogative and declarative prosody.

Contains narrative text with long quoted dialogues of emotion intensification, followed by contrasting emotions.

Contains vocal interjections (Uhh, Hmmm), Onomatopoeia (Achoo, tick-tock, etc), Varied Emphasis markers (Capitalization, vowel elongation, syllable emphasis with hyphens), Pacing cues like ellipses (….) or punctuation (STOP.RIGHT.THERE), and stuttering (I-I-I d-didn’t, so so-so-so-sorry).

Covers 15 unique languages with idiomatically and prosodically rich phrases placed in between english text.

Covers complex text with garden-path sentences, deep nested clauses with centre embeddings, homographs, and other forms of syntactic complexity.

Texts with emails, phone numbers, URLs, Street Adresses, Location references, STEM equations, units and notations, Abbreviations-Both initialisms (pronounced letter by letter) and acronyms (pronounced as word), and tongue twisters.

Synthesizing speech for all $1{,}645$ benchmark utterances results in approximately $420$ minutes (or $7$ hours) of audio per TTS system. Evaluating this volume of audio through human raters is both time-consuming and resource-intensive, with limited reproducibility, and the need for specialized linguistic knowledge. To address these limitations, we employ Large-Audio-Language Models (LALMs) as automatic judges. Our benchmark specifically targets aspects of speech synthesis-such as prosody, pausing, expressiveness, and pronunciation-that are not adequately captured by traditional metrics like Word Error Rate (WER) or MOS-based quality estimators. Accurately assessing these dimensions requires a general-purpose, high-capacity audio understanding model.

We choose Gemini 2.5 Pro as our primary LALM-based judge due to its strong performance on established audio reasoning benchmarks such as MMAU (mmaumassiv, ) (See Appendix B for performance comparison of different LALMs on MMAU). Notably, it leverages inference-time scaling deepseekai2025deepseekr1incentivizingreasoningcapability ; liu2025inferencetimescalinggeneralistreward before producing outputs, which aligns well with the complexity of our evaluation tasks.

To evaluate a candidate TTS system $T_{i}$, we compare it against a strong reference system $T_{j}$, chosen to have low WER to ensure high-fidelity synthesis of the benchmark utterances. For each evaluation instance, both systems generate speech for the same input, and the outputs are randomly assigned as $T_{1}$ and $T_{2}$ to avoid positional bias. The LALM judge is provided with the original text, the associated category label, and a structured evaluation prompt that includes the target evaluation dimension (e.g., prosody, emotion), scoring rubric, and detailed category-specific reasoning guidelines. The model is then presented with the audio from $T_{1}$, followed by a separation marker, and then the audio from $T_{2}$.

The LALM returns a structured json response containing natural language justifications for the performance of each system, a comparative analysis highlighting key differences-annotated as either subtle or significant-a scalar score in the range $[0,3]$ for each system, and a final winner label: $0$ for a tie, $1$ if $T_{1}$ is preferred, and $2$ for $T_{2}$. The prompt is designed to elicit chain-of-thought reasoning with time-stamp based analysis, and encourages the model to resolve borderline cases by articulating fine-grained distinctions and predict human-based preferences. The full judger prompts used for each category are shared in the Appendix C.3.

We adopt a win-rate-based metric to summarize performance. Let $W(T_{i})$ denote the win-rate of system $T_{i}$ relative to the baseline $T_{j}$. This is computed as:

where $\text{index}_{i}\in{1,2}$ corresponds to the randomized label assigned to $T_{i}$, and $n$ is the total number of comparisons. A score of $0.5$ reflects parity with the baseline, while deviations indicate relative superiority or inferiority.

This evaluation protocol enables robust, interpretable, and reproducible TTS comparison at scale. Unlike human raters, the LALM offers consistent judgments across multilingual and prosodically rich utterances, and its outputs include timestamp-grounded rationales that support fine-grained diagnostic analysis as evidenced by examples provided in the Appendix D. Our experiments in Section 4.6.2 show that the judge-based win-rate has high correlation with human preference as well.

While Gemini 2.5 Pro achieves the highest performance on the MMAU mmaumassiv benchmark for audio understanding, we conducted an ablation study to assess how evaluation outcomes vary across different LALM judger models, both proprietary and open-source. Using identical audio inputs from candidate TTS systems, we varied the judger model across four closed-source and one open-source alternative. Results are shown in Table 2.

Our analysis reveals that Qwen 2.5 Omni performs poorly in the judging role, frequently producing parsing errors and yielding win-rates near 50% across the board-indicative of near-random behavior. In contrast, the remaining judger models (Gemini 2.0 Flash, Gemini 2.5 Flash, Gemini 2.5 Pro, GPT-4o-mini-audio, and GPT-4o-audio) exhibit strong agreement in their relative rankings, despite differences in absolute scores. This alignment is quantified by a high Kendall’s W coefficient of concordance ($W=0.97$), indicating near-perfect inter-model consistency and further validating the robustness of our evaluation framework.

Most TTS models are tied to specific voices-either through fine-tuning or voice cloning-except for a few, such as Hume.AI and Sesame1B, which generate different voice for different utterances. To examine the impact of voice identity on performance, we measure the category-wise standard deviation in win-rate across multiple voices for four models: GPT-4o-mini-tts (6 voices: Alloy, Ballad, Ash, Coral, Nova, Onyx), Deepgram Aura-2 (6 voices: Thalia-en, Andromeda-en, Helena-en, Apollo-en, Arcas-en, Aries-en), Orpheus-TTS (Tara, Leah, Jess, Leo, Dan, Mia), and Qwen 2.5 Omni (2 voices: Chelsie and Ethan). Results are shown in Figure 4(a). We find that Emotions and Paralinguistics exhibit the highest sensitivity to voice variation, reflected in elevated standard deviations. This is consistent with the fact that voice fine-tuning often emphasizes expressive rendering, which these categories demand. In contrast, Pronunciation shows the least variance across voices, as it depends more on the inherent ability of the model and not the voice characteristics, other categories also show low variance generally.

The main challenge of the complex pronunciation category lies in parsing uncommon characters and their groups, something that can be made easier by using Text Normalization(TN) techniques prior to sending the text to the TTS model. To this end, we do an ablation measuring the change in win-rate for various TN techniques. We also add the data point corresponding to an LLM (GPT-4.1-mini) acting as the TN, the results are in Table 3(a).

We note that basic TN techniques do not always improve model performance on our benchmark and can make it worse. For instance, WeText (wetext-tn, ) converts ’$1,890.125375’ to ‘one thousand eight hundred and ninety point one dollars twenty five thousand three hundred and seventy five’, which harms TTS quality. Similarly, ’0’ is sometimes normalized to the informal ’oh’, which is not preferred in formal or decimal contexts. ’SQL’ was correctly normalized to ’S Q L’, but the baseline’s pronunciation ’Sequel’ was preferred. Using an LLM for TN resolves many of these issues and significantly improves win-rate, though some errors persist with the basic prompt that we used. We include more examples in the Appendix E.