multilingual-speech-benchmark

# Multilingual Speech Benchmark for Zero-Shot TTS A benchmark for evaluating **zero-shot text-to-speech (voice cloning)** across six language variants, built on the [Seed-TTS-eval](https://github.com/BytedanceSpeech/seed-tts-eval) protocol and extended with controlled length distributions, phonetic coverage guarantees, and **human toplines for every metric**. **8,200 examples across 6 language variants**, each with a reference prompt, a target text, ground-truth audio, and a second reference clip of the prompt speaker. | Subset | Examples | Speakers | Diphone coverage | Female voices | WER topline | SIM topline | |---|---|---|---|---|---|---| | `en-US` | 1,500 | 1,162 | 100% | 45% | 8.06% | 0.932 | | `es-ES` | 1,500 | 722 | 100% | 38% | 4.85% | 0.944 | | `es-MX` | 1,500 | 929 | 100% | 37% | 6.54% | 0.946 | | `nl-NL` | 1,500 | 469 | 100% | 18% | 4.07% | 0.929 | | `pt-BR` | 1,500 | 247 | 100% | 9% | 8.23% | 0.934 | | `ky` | 700 | 181 | 100% | 17% | 11.04% | 0.921 | --- ## 1. Why this benchmark exists Most zero-shot TTS evaluation reports two numbers — WER of an ASR system on the synthesized audio, and speaker similarity (SIM) between the synthesis and the reference prompt. Both are close to meaningless in isolation: - **A WER of 8% means nothing without knowing what the ASR scores on real human speech of the same texts.** In this dataset, Whisper-large-v3 scores 4.07% WER on genuine Dutch recordings and 8.06% on genuine US English ones. A model at 9% WER is near-human on English and roughly twice as bad as human on Dutch. The same number, two opposite conclusions. - **A SIM of 0.90 means nothing without knowing the ceiling.** Two different recordings *of the same living person* score 0.92–0.95 here. A model at 0.90 is therefore close to the physical ceiling of the metric, not mediocre. Every subset in this dataset ships both toplines, computed on the same audio, with the same models, under the same normalization. The second gap this addresses is **coverage**. Randomly sampling 1,500 utterances from Common Voice yields a set dominated by a handful of prolific speakers, skewed in length, and blind to large parts of the phonetic inventory. Selection here is a constrained optimization, not `sample()` — see [§5](#5-example-selection-s3). --- ## 2. Quick start ```python from datasets import load_dataset ds = load_dataset("nineninesix/multilingual-speech-benchmark", "es-MX", split="main") ex = ds[0] ex["prompt_audio"] # reference for cloning: 16 kHz, VAD-trimmed, peak −1 dBFS ex["prompt_text"] # its transcript (required by F5-TTS, CosyVoice, XTTS, …) ex["text"] # the text the model must speak ex["gt_audio"] # a real human recording of that text → WER topline ex["sim_ref_audio"] # a second clip of the prompt speaker → SIM topline ``` ### Seed-TTS-eval compatibility The dataset is designed to be consumable by the **unmodified** `seed-tts-eval` scripts. Export the flat layout they expect with: ```python import soundfile as sf from pathlib import Path from datasets import load_dataset lang, out = "es-MX", Path("seed-tts-eval/es-MX") for sub in ("prompt-wavs", "gt-wavs", "sim-refs"): (out / sub).mkdir(parents=True, exist_ok=True) lines = [] for ex in load_dataset("nineninesix/multilingual-speech-benchmark", lang, split="main"): u = ex["utt"] sf.write(out / "prompt-wavs" / f"{u}.wav", ex["prompt_audio"]["array"], 16000) gt = "" if ex["has_gt"]: sf.write(out / "gt-wavs" / f"{u}.wav", ex["gt_audio"]["array"], 16000) gt = f"gt-wavs/{u}.wav" if ex["has_sim_ref"]: sf.write(out / "sim-refs" / f"{u}.wav", ex["sim_ref_audio"]["array"], 16000) lines.append("|".join([u, ex["prompt_text"], f"prompt-wavs/{u}.wav", ex["text"], gt])) (out / "meta.lst").write_text("\n".join(lines) + "\n") ``` This produces the canonical five-field format, read unchanged by `get_wav_res_ref_text.py`: ``` utt|prompt_text|prompt_wav|infer_text|infer_wav ``` ### Evaluation protocol 1. For each example, synthesize `text` conditioned on `prompt_audio` (+ `prompt_text` if your model needs it). Save as `{utt}.wav`. 2. **WER** — transcribe the synthesis, normalize both sides identically (lowercase, strip punctuation, expand digits, **keep diacritics**), compute WER/CER. 3. **SIM** — cosine between speaker embeddings of the synthesis and `prompt_audio`. 4. Report against the toplines in [§6](#6-human-toplines). --- ## 3. Source data All audio and text originate from **Common Voice 17.0** (CC0-1.0). Mozilla removed the Common Voice datasets from the Hugging Face Hub in October 2025; the `mozilla-foundation/common_voice_*` repositories are now empty. This build uses the community mirror [`fsicoli/common_voice_17_0`](https://huggingface.co/datasets/fsicoli/common_voice_17_0), which is not gated and covers 122 locales. **CV 17 was chosen over newer releases deliberately.** Dev/test splits in Common Voice are size-capped and barely grow between releases, while `pt/train` *collapsed* from 143,181 rows in CV17 to 22,924 in CV22 — the splitting algorithm changed. Newer is not larger for the variants used here. ### 3.1 Regional variants are not locales There are no `es-MX`, `pt-BR` or `nl-NL` locales in Common Voice. Regional variants live in free-text, self-declared metadata columns: | Variant | Column | Label | |---|---|---| | es-MX | `accents` | `México` | | es-ES | `accents` | `España: Norte peninsular (…)` + `España: Centro-Sur peninsular (…)` | | pt-BR | `variant` | `Portuguese (Brasil)` | | nl-NL | `accents` | `Nederlands Nederlands` | | en-US | `accents` | `United States English` | | ky | — | none; Kyrgyz has no variant labels at all | **Parsing these labels correctly is harder than it looks, and getting it wrong silently corrupts the dataset.** Three compounding traps: 1. The field is **multi-valued**, comma-separated: `Nederlands Nederlands,Amsterdams`. 2. **The labels themselves contain commas**, inside parentheses: `España: Norte peninsular (Asturias, Castilla y León, Cantabria, …)`. A naive `split(",")` shreds this into fragments and the es-ES slice cannot be assembled at all. 3. **Substring matching produces false positives.** The Caribbean label contains `Costa del golfo de México`, so `contains("México")` pulls in Caribbean Spanish. The working approach splits on commas **at parenthesis depth zero**, then matches components exactly. A residual trap remains and is documented for anyone extending this work: some Spanish labels enumerate countries *without* parentheses (`Andino-Pacífico: Colombia, Perú, Ecuador, …`), and those cannot be recovered by any splitting rule — they require a dictionary of known labels. This does not affect the six variants here. ### 3.2 What "es-ES" means here `España` is not one label but a family. The composition of the slice is a deliberate phonetic decision, not a convenience: | Label | Clips | Speakers | Included | |---|---|---|---| | `España: Norte peninsular (…)` | 31,968 | 436 | **yes** | | `España: Centro-Sur peninsular (Madrid, Toledo, Castilla-La Mancha)` | 9,839 | 332 | **yes** | | `España: Sur peninsular (Andalucia, Extremadura, Murcia)` | 40,327 | 180 | no | | `España: Islas Canarias` | 2,161 | 140 | no | | Cataluña / Valenciana / Galicia / … | < 50 each | 1–2 | no | `Sur peninsular` is the **largest** slice but is excluded for two independent reasons. Phonetically, Andalusian Spanish has *seseo*/*ceceo* and /s/ aspiration, which places it closer to Latin American Spanish than to the Castilian norm — including it would blur the very contrast between `es-ES` and `es-MX` that having both subsets is meant to test. Structurally, **82% of that slice is a single speaker**. Canarian Spanish is excluded for the same phonetic reason (seseo). The resulting contrast is verifiable in the data: **/θ/ appears in the `es-ES` phoneme inventory and is absent from `es-MX`.** --- ## 4. Quality control (S2) Common Voice is crowdsourced; "validated" means only that two listeners clicked approve. The QC stage here is deliberately **narrower than usual**, and the reasoning matters for interpreting the data. ### 4.1 What is checked - **Decode sanity** — true sample rate, channel count, clipping rate, DC offset. - **Bandwidth** — energy above 5 kHz, in dB. Clips upsampled from 8 kHz are unusable as cloning references and are removed (they are ~1–2% of the pool). - **VAD (Silero)** — leading/trailing silence trimmed; speech ratio after trimming; recordings with an internal pause > 0.25 s ("started, stumbled, re-recorded") rejected. - **Speaker consistency** — WavLM-SV embeddings of all clips of a `client_id`; clips beyond μ−2σ from the speaker centroid removed. ### 4.2 What is deliberately *not* checked, and why **Noise metrics (DNSMOS, WADA-SNR) are not used for filtering.** References in this benchmark are intended to be passed through speech-restoration models downstream, which makes pre-restoration noise a poor selection criterion. This decision was then **tested rather than assumed**. DNSMOS P.835 was computed on all 8,200 references and correlated against the SIM topline: | | en-US | es-ES | es-MX | nl-NL | pt-BR | ky | |---|---|---|---|---|---|---| | Pearson *r* | 0.079 | 0.049 | 0.124 | 0.064 | 0.114 | 0.091 | | Spearman ρ | 0.070 | 0.082 | 0.123 | 0.087 | 0.031 | 0.033 | Reference quality explains **less than 1.6% of SIM variance**. Splitting SIM by DNSMOS tercile gives a spread of 0.5–1.7 points on a 0.92–0.95 scale, and the ordering is not even monotonic for `es-ES` and `nl-NL`. Reference quality is **not a confounder** for speaker similarity in this dataset. > **Caveat.** This was measured on the *human* SIM topline, where noise affects both > embeddings and partially cancels. Whether a noisy reference degrades cloning by an > actual TTS model — which conditions on that reference — is a different question that > only a model run can answer. DNSMOS values are shipped as reportable columns (`qc_dnsmos_*`). Corpus-wide OVRL is 2.68–2.81 with p05 ≈ 1.9–2.2 — uniformly mediocre, as expected for crowdsourced audio. **ASR verification of prompts is also not performed** (`qc_asr_cer` is always null). The residual risk is stated plainly: `prompt_text` is fed to the model as conditioning, so a CV transcript that does not match its audio injects noise into every model's score equally. --- ## 5. Example selection (S3) ### 5.1 Prompts and targets are decoupled The single most consequential design decision. It is tempting to require that the target text be spoken by the prompt speaker, so that ground-truth audio is "paired". That requirement conflates two independent needs: | Topline | What it actually requires | Prompt speaker relevant? | |---|---|---| | **WER (human)** | any real recording of `text` | **no** | | **SIM (human)** | a second clip of the prompt speaker, **any text** | yes, but the text is irrelevant | Decoupling them means target texts are chosen freely from the entire language pool, which is what makes the length distribution and phonetic coverage below achievable. Only the language and regional variant must match. This does not break format compatibility: the stock `seed-tts-eval` scripts never read the fifth field — SIM uses `prompt_wav`, WER uses `infer_text`. Consequently, `gt_audio` may be spoken by a different person than `prompt_audio`; `gt_same_speaker` marks the cases where they coincide, and only there is `SIM(synthesis, gt_audio)` meaningful. ### 5.2 The optimization Texts are selected by **lazy greedy diphone set-cover** (the coverage function is submodular, giving a 1−1/e approximation), subject to hard constraints: ``` maximize Σ gain_diphone(i) + w · [split_origin(i) ∈ {dev, test}] subject to |S| = N examples_per_speaker(s) ≤ cap_lang length_bin_distribution = target ± 2% female_voice_share ≥ target_f unique sentences prompt duration stratified ``` Result: **100% of the pool's diphone inventory is covered in all six subsets**, with length bins matching target exactly. `w · [dev/test]` is a soft preference for held-out clips. Because the target model was not trained on Common Voice, the full pool was usable; the preference costs nothing and leaves users whose models *did* train on CV a usable held-out subset — 44–95% of prompts and 13–96% of texts come from `dev`/`test`, recorded per example in `*_split_origin`. ### 5.3 Per-language parameters Nothing here is uniform across languages, because the corpora are not: | | en-US | es-ES | es-MX | nl-NL | pt-BR | ky | |---|---|---|---|---|---|---| | N | 1,500 | 1,500 | 1,500 | 1,500 | 1,500 | **700** | | Speakers available | 1,736 | 722 | 929 | 469 | 247 | 181 | | `cap` per speaker | 2 | 3 | 2 | 4 | **7** | 4 | | Length bins 3-5 / 6-9 / 10-12 / 13-16 / 17+ | 15/30/30/23/2% | 15/30/30/25/0% | 15/30/30/25/0% | 15/30/30/23/2% | **35/35/20/10/0%** | **35/55/10/0/0%** | `pt-BR` bins are shifted left because its median utterance is 6 words. `ky` has a **hard maximum of 13 words in the entire language**, so the two longest bins are physically empty. `pt-BR` uses `cap=7` because only 247 speakers survived QC and 247 × 6 < 1,500. ### 5.4 Phonemization espeak-ng 1.51 via `phonemizer`, using `es-419` for es-MX (**not** `es`), `es` for es-ES, `pt-br`, `nl`, `en-us`, `ky`. **The Kyrgyz voice required special handling.** It emits X-SAMPA-like ASCII rather than IPA — `ө`→`oe`, `ч`→`tS`, `ж`→`dZ`, `ң`→`N`, `х`→`X`, length as `:`. Naive character-level tokenization would split `tS` into `t`+`S` and build the diphone inventory on phonemes that do not exist. Tokenization uses longest-match against a symbol table. More seriously, on tokens it cannot read (isolated letters, Latin script) espeak **silently switches to the English voice** and wraps the output in `(en)…(ky)`, leaking English phonemes `ɹ ɪ ə` into the Kyrgyz inventory. Such texts are **rejected**, not repaired — 3.0% of the Kyrgyz pool. --- ## 6. Human toplines Computed on `gt_audio` (WER/CER) and on `sim_ref_audio` vs `prompt_audio` (SIM). | Subset | ASR | WER | CER | Exact match | SIM topline | |---|---|---|---|---|---| | nl-NL | Whisper-large-v3 | 4.07% | 1.18% | 76% | 0.929 | | es-ES | Whisper-large-v3 | 4.85% | 1.73% | 73% | 0.944 | | es-MX | Whisper-large-v3 | 6.54% | 2.35% | 68% | 0.946 | | pt-BR | Whisper-large-v3 | 8.23% | 2.43% | 71% | 0.934 | | en-US | Whisper-large-v3 | 8.06% | 3.11% | 60% | 0.932 | | ky | GigaAM-Multilingual | 11.04% | 3.36% | 66% | 0.921 | Whisper is unusable for Kyrgyz; [`ai-sage/GigaAM-Multilingual`](https://huggingface.co/ai-sage/GigaAM-Multilingual) (revision `ctc`) is used instead, which lists `ky` among its languages. Decoding is greedy (`num_beams=1`) with a token cap, **identically for all six subsets**. This is not only a cost choice: with beam search, Whisper occasionally enters a long generation on short clips and the KV cache grows until it exhausts GPU memory. An earlier revision of this card reported nl-NL at 3.80% and pt-BR at 7.75%; those two had been decoded with beam search while the rest used greedy, which made the numbers not strictly comparable across languages. All six have been recomputed under identical settings. **Kyrgyz exceeds the 10% threshold** at which a second ASR cross-check becomes advisable — it is currently unknown how much of that 11% is Kyrgyz phonetics being genuinely hard versus a systematic property of GigaAM. ### SIM topline vs reference duration The reason `prompt_dur` is a first-class column: | Subset | < 3.7 s | 3.7–4.5 s | ≥ 4.5 s | |---|---|---|---| | en-US | 0.923 | 0.928 | 0.937 | | es-ES | 0.932 | 0.948 | 0.946 | | es-MX | 0.938 | 0.946 | 0.949 | | nl-NL | 0.925 | 0.931 | 0.937 | | pt-BR | 0.926 | 0.949 | 0.940 | | ky | 0.917 | 0.932 | 0.914 | The dependence is monotonic and weak for five subsets (≈ +1–1.5 points per 1.5 s). For `ky` it is **non-monotonic**, which may be noise at n=103 in the top bin. --- ## 7. Data fields <details> <summary>Full schema (58 columns)</summary> **Identification** — `utt`, `lang`, `subset`, `source`, `cv_version` **Prompt** — `prompt_audio` (16 kHz, VAD-trimmed, peak −1 dBFS), `prompt_audio_orig` (original sample rate), `prompt_text`, `prompt_dur` (**after trimming**), `prompt_sr_orig`, `prompt_dur_bin`, `prompt_cv_path`, `prompt_split_origin` **Prompt speaker** — `speaker_id` (hashed), `speaker_gender`, `speaker_gender_source`, `speaker_age`, `speaker_accent_label` (raw CV label including secondary accents, e.g. `Nederlands Nederlands,Amsterdams`), `speaker_n_in_subset` **Target text** — `text`, `text_norm`, `n_words`, `n_chars`, `len_bin`, `phones` (IPA), `n_phones`, `punct_type`, `text_cv_path`, `text_split_origin` **Ground truth (WER topline)** — `gt_audio`, `has_gt`, `gt_dur`, `gt_speaker_id`, `gt_same_speaker`, `gt_gender` **SIM reference (SIM topline)** — `sim_ref_audio`, `has_sim_ref`, `sim_ref_text`, `sim_ref_dur` **QC** — `qc_dnsmos_ovrl/_sig/_bak`, `qc_vad_speech_ratio`, `qc_lead_sil`, `qc_trail_sil`, `qc_clip_rate`, `qc_bandwidth_hz`, `qc_spk_centroid_dist`. `qc_snr_db` and `qc_asr_cer` are **always null in v1** — not computed, see [§4.2](#42-what-is-deliberately-not-checked-and-why). **Categories** — `category`, `subcategory`, `tags`, `difficulty`, `has_digit`, `has_abbrev`, `has_foreign`, `notes`. All examples in `main` are `category="general"`; these fields exist for the planned hard set. </details> **`n_syllables` is intentionally absent.** No syllable-counting rule was stable across these languages: merging adjacent vowels breaks on Spanish hiatus (`¿Qué día de la semana es?` → 7 instead of 9), while counting every vowel breaks on diphthongs (`…María Eugenia…` → 26 instead of 21). Its correlation with `n_phones` is 0.97–0.985, so it carried no independent information. For speech rate, use `n_phones / gt_dur`. --- ## 8. Limitations Read this section before reporting numbers from this benchmark. 1. **Gender balance is capped by the corpus, not by the selection.** Female-speaker shares are 45% / 38% / 37% / 18% / 17% / 9% (en-US / es-ES / es-MX / nl-NL / ky / pt-BR). `pt-BR` has **19 female speakers in the entire variant**; a balanced Brazilian Portuguese benchmark cannot be built from Common Voice at all. Per-gender breakdowns for `pt-BR` and `ky` are statistically thin. 2. **Accent labels are self-declared** and unverified. A speaker labelled `México` is someone who selected that option; no accent classifier was run. 3. **Utterance length is capped by the source.** Common Voice sentences top out at 15–18 words (39 for en-US, 13 for ky). **Long-form synthesis is not tested.** 4. **Prosody is barely tested.** Common Voice texts are overwhelmingly declarative: questions are 0.2–9% of the pool and exclamations 0.3–1.8%, depending on language. The selection takes everything available up to a 10% cap, but a genuine prosody evaluation requires authored texts. 5. **No digits, abbreviations, or acronyms.** Common Voice contains **0%** texts with digits — they were removed at sentence-collection time. Text normalization, the most common failure mode of production TTS, is not measured here at all. 6. **`ky` is smaller by necessity, not by choice.** The entire Kyrgyz corpus with available audio is 5,016 clips from 211 speakers, and CV22 is the same size — the mirror is exhausted. N=700 is the ceiling given a 4-example-per-speaker cap. Gender is undeclared for 44% of Kyrgyz speakers. 7. **`pt-BR` splits are broken upstream.** 97% of test sentences also appear in train, 648 speakers are shared between train and test, and **9,464 clips are physically duplicated across `train` and `dev`** (the same file, listed twice). Deduplication here is by `path`, not only by sentence. A "clean held-out set" does not exist for Brazilian Portuguese in CV17. 8. **SIM uses `microsoft/wavlm-base-plus-sv`**, not the `wavlm_large_finetune.pth` checkpoint of the original Seed-TTS-eval. Absolute SIM values are **not comparable** across checkpoints — compare only numbers computed with the same model. 9. **`sim_ref_audio` is missing for 26–39% of examples**, where the speaker had only one clip passing QC. Those examples remain valid; they simply do not contribute to the SIM topline. Filter on `has_sim_ref`. 10. **The hard set is not included in v1.** The schema reserves `subset="hard"` and a 13-category taxonomy, but authoring those texts — and validating the Kyrgyz ones with a native speaker — is future work. 11. **No TTS model has been run on this benchmark yet.** The toplines are computed and internally consistent, but the benchmark has not yet demonstrated that it produces a meaningful spread between real systems. --- ## 9. Reproduction The full pipeline is deterministic given the source corpus (fixed RNG seed 20260725): | Stage | What it does | |---|---| | **S0** | Download CV17 transcripts and audio shards; verify tar integrity against TSVs | | **S1** | Hard filters: variant label, votes, duration, alphabet, dedup by `path` / `sentence_id` / text; phonemization with language-switch rejection | | **S2** | Audio QC on prompt candidates; one prompt selected per speaker | | **S3** | Greedy diphone set-cover under length, speaker, gender and prosody constraints | | **S4** | Resample, trim, peak-normalize, package as parquet + `meta.lst` | Audio is read directly from the tar shards by byte offset (an index of 1.73M clips), so the 64 GB corpus never needs to be unpacked. --- ## 10. License and citation The source corpus, Common Voice 17.0, is **CC0-1.0**; this dataset inherits it. ```bibtex @misc{multilingual_speech_benchmark_2026, title = {Multilingual Speech Benchmark for Zero-Shot TTS}, author = {nineninesix}, year = {2026}, url = {https://huggingface.co/datasets/nineninesix/multilingual-speech-benchmark} } ``` Please also cite Common Voice: ```bibtex @inproceedings{commonvoice2020, title = {Common Voice: A Massively-Multilingual Speech Corpus}, author = {Ardila, Rosana and Branson, Megan and Davis, Kelly and Kohler, Michael and Meyer, Josh and Henretty, Michael and Morais, Reuben and Saunders, Lindsay and Tyers, Francis M. and Weber, Gregor}, booktitle = {Proceedings of LREC}, pages = {4218--4222}, year = {2020} } ``` ## References - Anastassiou et al., *Seed-TTS: A Family of High-Quality Versatile Speech Generation Models*, [arXiv:2406.02430](https://arxiv.org/abs/2406.02430) - Seed-TTS-eval protocol — https://github.com/BytedanceSpeech/seed-tts-eval - Radford et al., *Robust Speech Recognition via Large-Scale Weak Supervision* (Whisper), [arXiv:2212.04356](https://arxiv.org/abs/2212.04356) - Chen et al., *WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing*, [arXiv:2110.13900](https://arxiv.org/abs/2110.13900) - Reddy et al., *DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric*, [arXiv:2110.01763](https://arxiv.org/abs/2110.01763) - GigaAM — [`ai-sage/GigaAM-Multilingual`](https://huggingface.co/ai-sage/GigaAM-Multilingual), [arXiv:2607.10371](https://arxiv.org/abs/2607.10371) - Silero VAD — https://github.com/snakers4/silero-vad - espeak-ng — https://github.com/espeak-ng/espeak-ng - Common Voice variant naming — https://common-voice.github.io/community-playbook/sub_pages/Lang_Variant.html