Chinese Speech Prosody Benchmark

How to model and estimate speech prosody is considered a challenging task in understanding and generating natural speech. We introduce the Mandarin Speech Prosody Benchmark (MSPB), a linguistically grounded dataset for evaluating Speech Large Language Models (Speech LLMs) in Mandarin. MSPB comprises eight tasks covering crucial prosodic features and their interactions with syntax, semantics, and pragmatics. All MSPB items, designed per Mandarin linguistic principles and validated by experts, were phonetically recorded and verified. We evaluated six Speech LLMs (GPT-4o, Gemini-1.5-Pro, Gemini-2-Flash, Qwen2-Audio-7B-Instruct, GLM-4-Voice, MiniCPM-o 2.6). Although some models perform well with context-rich cues (e.g., irony), they generally struggle with subtle prosodic variations (e.g., focus marking) and underperform humans. MSPB provides a valuable tool for assessing and advancing prosodic comprehension, highlighting the importance of enhanced prosodic integration in future research. This release includes the MSPB dataset, comprising questions, corresponding audio files, and ground-truth answers, along with an example evaluation script for benchmarking Speech LLMs. Due to encoding issues with Zenodo’s online ZIP file viewer, some filenames containing Chinese characters may appear incorrectly on the web page. English translations and explanations for these filenames are provided in the README.