IndicSpeechBench

# SpeechArenaBench **SpeechArenaBench** is a large-scale human-preference dataset for evaluating multilingual Text-to-Speech (TTS) systems across 10 Indian languages. It accompanies the paper *"Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages"* (accepted to Interspeech 2026), and contains the full benchmark sentences, generated audio, and crowd-sourced pairwise preference judgments collected from native raters. The dataset is built around a **controlled, multidimensional pairwise evaluation framework**: for each comparison, two anonymized TTS systems synthesize the same sentence, and raters provide both a holistic overall preference and granular ratings across six perceptual axes — intelligibility, expressiveness, voice quality, liveliness, hallucinations, and presence of noise. ## Dataset Summary - **5,357** carefully curated and quality-checked evaluation sentences - **10 Indian languages**: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Tamil, Telugu, and Urdu - **16 deployment-relevant domains** (e.g. news, conversation, STEM content, tongue twisters, stress tests) - **7 TTS systems evaluated**: Gemini 2.5 Pro TTS, ElevenLabs v3, Sonic 3, Bulbul v3 Beta, Speech 2.8 HD, GPT-4o-mini TTS, and Indic F5 - **120K+ pairwise comparisons** collected from **1,900+ vetted native raters** across 22 Indian states - Three structured input subsets per language: **Normalized**, **Symbolic**, and **Code-mixed** text, reflecting real-world deployment conditions including intra-sentential English insertion, transliteration, and mixed-script usage - Six **fine-grained perceptual axes** rated independently of the overall preference, enabling diagnostic (not just aggregate) analysis of model behavior This makes SpeechArenaBench, to our knowledge, one of the largest controlled human-evaluation resources for multilingual and code-mixed TTS, and the first at this scale to combine pairwise overall preference with multi-axis perceptual diagnostics for Indic languages. ## Dataset Structure The dataset is organized as a Hugging Face `DatasetDict`, keyed by language code (ISO 639-1): ```python from datasets import load_dataset ds = load_dataset("ai4bharat/SpeechArenaBench", "hi") # Hindi subset ``` Available configs/splits: `bn`, `gu`, `hi`, `kn`, `ml`, `mr`, `or`, `ta`, `te`, `ur` Each language subset contains the evaluation sentences, the corresponding synthesized audio from the 7 TTS systems, and the rater annotations collected for that language. ### Fields (per comparison record) | Field | Description | |---|---| | `sentence` / `text` | The source evaluation sentence (native script, with code-mixing/symbolic content as applicable) | | `domain` | One of 16 deployment-relevant domains the sentence was drawn from | | `subset` | One of `normalized`, `symbolic`, or `codemixed` | | `model_a`, `model_b` | The two anonymized TTS systems being compared | | `audio_a`, `audio_b` | Synthesized audio (`Audio` feature) for each system | | `overall_preference` | Rater's holistic choice: `Model A`, `Model B`, `Both Good`, or `Both Bad` | | `intelligibility`, `expressiveness`, `voice_quality`, `liveliness`, `hallucinations`, `noise` | Per-axis ratings, collected after the overall preference is locked in | | `rater_id` | Anonymized rater identifier (salted-hash) | | `language` | Language code for the comparison | ## Annotation Protocol Ratings were collected using a strict **two-phase workflow** to avoid post-hoc rationalization: 1. **Phase 1 — Overall preference:** Raters listen to two anonymized, randomized audio samples (Model A / Model B) generated from the same prompt and submit a single holistic judgment — *Model A*, *Model B*, *Both Good*, or *Both Bad*. This choice is then permanently locked. 2. **Phase 2 — Granular axes:** The same audio pair is then rated independently across the six perceptual axes (Table below), on the same comparison scale. | Axis | What it captures | |---|---| | Intelligibility | Clarity and correctness of pronunciation, including native and code-mixed words | | Expressiveness | Appropriateness of prosody, intonation, and emotional delivery | | Voice Quality | Naturalness and human-likeness of the voice, including timbre and absence of artifacts | | Liveliness | Energy and pacing — engaging vs. monotonous delivery | | Hallucinations | Fidelity to input text — penalizes skipped words or unintended sounds | | Presence of Noise | Background artifacts or signal distortions such as hiss, clicks, or static | Each rater evaluated 150 randomly sampled sentences. Raters were recruited through a multi-stage screening and training process (auditory discrimination test, justification of choices against the perceptual rubric, and guideline training) before being given access to the annotation interface. All participants gave informed consent under an internally reviewed study protocol and were compensated at standard industry rates. ## Benchmark Construction Sentences were sourced natively and via translation, covering 16 deployment-relevant domains, and supplemented with edge-case sentences (tongue twisters, extreme repetitions, dense STEM content) and 100 expressive utterances for prosody/emotion testing. All sentences underwent quality assurance by in-house native-language experts for linguistic accuracy, fluency, and domain terminology before inclusion. Three structured subsets probe different real-world input conditions: - **Normalized:** numerals, equations, and acronyms fully verbalized - **Symbolic:** raw numerals, formulas, and operators retained - **Code-mixed:** intra-sentential English insertions, transliteration-based mixing, and mixed-script sentences ## Intended Uses - Training and validating automatic TTS evaluation metrics (e.g. learned preference predictors) against large-scale human judgments - Benchmarking new TTS systems against the released leaderboard using Bradley–Terry-style modeling - Studying which perceptual factors (expressiveness, intelligibility, robustness, etc.) drive human preference in multilingual and code-mixed speech - Research on evaluation reliability and sample efficiency for human-in-the-loop speech evaluation ## Privacy & Ethics Rater identities are anonymized; the `rater_id` field is a salted cryptographic hash and cannot be linked back to individual annotators. The study protocol underwent internal ethics review, and all raters provided informed consent and were compensated at standard industry rates. ## Citation If you use this dataset, please cite the accompanying paper: ```bibtex @misc{anand2026preferencesvoicefirstnationlargescale, title={Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages}, author={Srija Anand and Ashwin Sankar and Ishvinder Sethi and Aaditya Pareek and Kartik Rajput and Gaurav Yadav and Nikhil Narasimhan and Adish Pandya and Deepon Halder and Mohammed Safi Ur Rahman Khan and Praveen S V and Shobhit Banga and Mitesh M Khapra}, year={2026}, eprint={2604.21481}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2604.21481}, } ``` ## License Released under the MIT License. ## Generative AI Use Disclosure As disclosed in the accompanying paper, generative AI tools were used solely for language polishing during manuscript preparation. No generative AI system was used to produce experimental results, analyses, or scientific conclusions in the underlying study.