InstructTTS-Eval

# InstructTTSEval InstructTTSEval is a comprehensive benchmark designed to evaluate Text-to-Speech (TTS) systems' ability to follow complex natural-language style instructions. The dataset provides a hierarchical evaluation framework with three progressively challenging tasks that test both low-level acoustic control and high-level style generalization capabilities. - **Github Repository:** [https://github.com/KexinHUANG19/InstructTTSEval](https://github.com/KexinHUANG19/InstructTTSEval) - **Paper:** [InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems](https://arxiv.org/pdf/2506.16381) ## Dataset Structure The dataset consists of two splits stored as Parquet files with embedded audio: - `en`: English examples (1,000 samples) - `zh`: Chinese examples (1,000 samples) Each example contains: - `id`: Unique identifier - `text`: Text to be synthesized - `APS`: Acoustic-Parameter Specification - detailed acoustic control instructions - `DSD`: Descriptive-Style Directive - high-level style descriptions - `RP`: Role-Play instruction - contextual scenarios and character descriptions - `reference_audio`: Reference audio (16kHz WAV, embedded in Parquet) ## Tasks 1. **Acoustic-Parameter Specification (APS)**: Tests fine-grained control over acoustic properties like pitch, speed, volume, and timbre. 2. **Descriptive-Style Directive (DSD)**: Evaluates the ability to interpret and execute high-level style descriptions. 3. **Role-Play (RP)**: Assesses contextual understanding and style adaptation based on character or scenario descriptions. ## Loading Examples ```python from datasets import load_dataset # Load English subset dataset_en = load_dataset("CaasiHUANG/InstructTTSEval", split="en") # Load Chinese subset dataset_zh = load_dataset("CaasiHUANG/InstructTTSEval", split="zh") # Load both languages dataset = load_dataset("CaasiHUANG/InstructTTSEval") english_data = dataset["en"] chinese_data = dataset["zh"] # Verify splits print("Available splits:", list(dataset.keys())) # Should show ['en', 'zh'] # Access audio data (automatically loaded from embedded Parquet) example = dataset_en[0] print(f"Text: {example['text']}") print(f"Audio sampling rate: {example['reference_audio']['sampling_rate']}") print(f"Audio array shape: {example['reference_audio']['array'].shape}") # Play audio (if using jupyter/colab) import IPython.display as ipd ipd.Audio(example['reference_audio']['array'], rate=example['reference_audio']['sampling_rate']) ``` ## Citation ```bibtex @misc{huang2025instructttsevalbenchmarkingcomplexnaturallanguage, title={InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems}, author={Kexin Huang and Qian Tu and Liwei Fan and Chenchen Yang and Dong Zhang and Shimin Li and Zhaoye Fei and Qinyuan Cheng and Xipeng Qiu}, year={2025}, eprint={2506.16381}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2506.16381}, } ```