Chinese Speech Prosody Benchmark

How to model and estimate speech prosody is considered a challenging task in understanding and generating natural speech. We introduce the Mandarin Speech Prosody Benchmark (MSPB), a linguistically grounded dataset for evaluating Speech Large Language Models (Speech LLMs) in Mandarin. MSPB comprises eight tasks covering crucial prosodic features and their interactions with syntax, semantics, and pragmatics. All MSPB items, designed per Mandarin linguistic principles and validated by experts, were phonetically recorded and verified. We evaluated six Speech LLMs (GPT-4o, Gemini-1.5-Pro, Gemini-2-Flash, Qwen2-Audio-7B-Instruct, GLM-4-Voice, MiniCPM-o 2.6). Although some models perform well with context-rich cues (e.g., irony), they generally struggle with subtle prosodic variations (e.g., focus marking) and underperform humans. MSPB provides a valuable tool for assessing and advancing prosodic comprehension, highlighting the importance of enhanced prosodic integration in future research. This release includes the MSPB dataset, comprising questions, corresponding audio files, and ground-truth answers, along with an example evaluation script for benchmarking Speech LLMs. Due to encoding issues with Zenodo’s online ZIP file viewer, some filenames containing Chinese characters may appear incorrectly on the web page. English translations and explanations for these filenames are provided in the README.在自然语音的理解与生成领域，语音韵律的建模与估计始终是一项极具挑战性的任务。我们推出了普通话语音韵律基准数据集（Mandarin Speech Prosody Benchmark，简称MSPB），这是一款基于语言学理论构建的数据集，用于评估普通话场景下的语音大语言模型（Speech Large Language Models，Speech LLMs）。MSPB涵盖八项任务，覆盖核心韵律特征及其与句法、语义、语用的交互关系。该数据集的所有条目均依据普通话语言学原则设计，经专家审核验证，并完成了语音录制与校验。我们针对六款语音大语言模型展开了评估实验，涉及模型包括GPT-4o、Gemini-1.5-Pro、Gemini-2-Flash、Qwen2-Audio-7B-Instruct、GLM-4-Voice以及MiniCPM-o 2.6。尽管部分模型在处理富含语境的提示信息（如反讽场景）时表现优异，但整体而言，它们在应对细微韵律变化（如焦点标记）时仍存在显著困难，性能不及人类。MSPB为韵律理解能力的评估与提升提供了宝贵工具，同时凸显了未来研究中强化韵律整合能力的重要性。本次发布包含MSPB数据集本体，涵盖问题文本、对应音频文件及真值答案，同时附带一套用于语音大语言模型基准测试的示例评估脚本。由于Zenodo在线ZIP文件查看器存在编码兼容问题，部分包含中文字符的文件名可能在网页上显示异常。针对此类文件名，我们已在README文件中提供了英文译名与详细说明。