遇见数据集

anonyemnlpauthor18/KMMAU

收藏
Hugging Face2026-05-25 更新2026-05-31 收录
官方服务:

资源简介:

--- dataset_info: - config_name: age features: - name: id dtype: string - name: audio dtype: audio - name: question dtype: string - name: answer dtype: string - name: original_dataset dtype: string - name: original_sample_id dtype: string - name: start_time dtype: float64 - name: end_time dtype: float64 splits: - name: test num_examples: 276 - config_name: fact_extraction features: - name: id dtype: string - name: audio dtype: audio - name: question dtype: string - name: answer dtype: string - name: original_dataset dtype: string - name: original_sample_id dtype: string - name: start_time dtype: float64 - name: end_time dtype: float64 splits: - name: test num_examples: 98 - config_name: gender features: - name: id dtype: string - name: audio dtype: audio - name: question dtype: string - name: answer dtype: string - name: original_dataset dtype: string - name: original_sample_id dtype: string - name: start_time dtype: float64 - name: end_time dtype: float64 splits: - name: test num_examples: 270 - config_name: general_counting features: - name: id dtype: string - name: audio dtype: audio - name: question dtype: string - name: answer dtype: string - name: original_dataset dtype: string - name: original_sample_id dtype: string - name: start_time dtype: float64 - name: end_time dtype: float64 splits: - name: test num_examples: 41 - config_name: number_of_speakers features: - name: id dtype: string - name: audio dtype: audio - name: question dtype: string - name: answer dtype: string - name: original_dataset dtype: string - name: original_sample_id dtype: string - name: start_time dtype: float64 - name: end_time dtype: float64 splits: - name: test num_examples: 100 - config_name: role_profession features: - name: id dtype: string - name: audio dtype: audio - name: question dtype: string - name: answer dtype: string - name: original_dataset dtype: string - name: original_sample_id dtype: string - name: start_time dtype: float64 - name: end_time dtype: float64 splits: - name: test num_examples: 100 - config_name: topic_summary features: - name: id dtype: string - name: audio dtype: audio - name: question dtype: string - name: answer dtype: string - name: original_dataset dtype: string - name: original_sample_id dtype: string - name: start_time dtype: float64 - name: end_time dtype: float64 splits: - name: test num_examples: 100 - config_name: word_frequency_counting features: - name: id dtype: string - name: audio dtype: audio - name: question dtype: string - name: answer dtype: string - name: original_dataset dtype: string - name: original_sample_id dtype: string - name: start_time dtype: float64 - name: end_time dtype: float64 splits: - name: test num_examples: 180 - config_name: word_order features: - name: id dtype: string - name: audio dtype: audio - name: question dtype: string - name: answer dtype: string - name: original_dataset dtype: string - name: original_sample_id dtype: string - name: start_time dtype: 'null' - name: end_time dtype: 'null' splits: - name: test num_examples: 1039 configs: - config_name: age data_files: - split: test path: age/test-* - config_name: fact_extraction data_files: - split: test path: fact_extraction/test-* - config_name: gender data_files: - split: test path: gender/test-* - config_name: general_counting data_files: - split: test path: general_counting/test-* - config_name: number_of_speakers data_files: - split: test path: number_of_speakers/test-* - config_name: role_profession data_files: - split: test path: role_profession/test-* - config_name: topic_summary data_files: - split: test path: topic_summary/test-* - config_name: word_frequency_counting data_files: - split: test path: word_frequency_counting/test-* - config_name: word_order data_files: - split: test path: word_order/test-* license: cc-by-nc-sa-4.0 task_categories: - automatic-speech-recognition - question-answering language: - ko pretty_name: KMMAU size_categories: - 1K<n<10K --- # KMMAU <p align="center"> <a href="#license"><img src="https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey?style=flat" alt="License"></a> </p> **KMMAU** is a Korean audio understanding benchmark constructed from naturally occurring Korean audio. It is part of a Korean speech benchmark suite for evaluating SpeechLMs together with KVoiceBench and KOpenAudioBench. ## Dataset Description KMMAU contains **2,204** Korean audio understanding samples built from KSS, KMSAV, and the Seoul Corpus rather than transferred English items. It evaluates both acoustic and contextual audio understanding capabilities using multiple-choice questions grounded in Korean audio. KMMAU is constructed from target-language ASR corpora using four construction methods selected by capability: 1. **Rule-based generation from speaker metadata** for acoustic attributes such as age, gender, and number of speakers. 2. **Rule-based generation from transcriptions** for lexical questions such as word order and word frequency. 3. **LLM-generated questions with human review** for semantic questions such as fact extraction and topic summary. 4. **Fully manual annotation** for holistic capabilities that require listening to the audio, such as general counting and role/profession. The benchmark contains 646 acoustic samples and 1,558 contextual samples. All KMMAU samples are multiple-choice: gender uses 2 choices, age uses 3 choices, and the remaining capabilities use 4 choices. ## Subsets The dataset consists of **2,204** samples across 9 capabilities: | Config | Category | Samples | Capability | |---|---|---:|---| | `age` | Acoustic | 276 | Speaker age group estimation | | `gender` | Acoustic | 270 | Speaker gender identification | | `number_of_speakers` | Acoustic | 100 | Number of speakers detection | | `word_order` | Contextual | 1,039 | Word order verification | | `word_frequency_counting` | Contextual | 180 | Word frequency counting in speech | | `fact_extraction` | Contextual | 98 | Factual information extraction from dialogue | | `topic_summary` | Contextual | 100 | Topic summarization | | `role_profession` | Contextual | 100 | Speaker role/profession identification | | `general_counting` | Contextual | 41 | General counting over audio content | ## Features - **id** (string): Unique sample identifier - **audio** (Audio): Korean speech audio - **question** (string): Question about the audio - **answer** (string): Reference answer - **original_dataset** (string): Source dataset name - **original_sample_id** (string): Sample identifier in the source dataset - **start_time** (float, optional): Audio segment start time in seconds - **end_time** (float, optional): Audio segment end time in seconds ## Usage ```python from datasets import load_dataset # Load a specific subset ds = load_dataset("anonyemnlpauthor18/KMMAU", "age", split="test") print(ds[0]) # Load all subsets subsets = [ "age", "fact_extraction", "gender", "general_counting", "number_of_speakers", "role_profession", "topic_summary", "word_frequency_counting", "word_order" ] for subset in subsets: ds = load_dataset("anonyemnlpauthor18/KMMAU", subset, split="test") print(f"{subset}: {len(ds)} samples") ``` ## Citation ```bibtex @article{korean_speech_benchmarks2026, title={KVoiceBench, KOpenAudioBench, and KMMAU: Agent-Driven Korean Speech Benchmarks for Evaluating SpeechLMs}, author={Anonymous Authors}, year={2026} } ``` ## License This dataset is released under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/). &copy; 2026 Anonymous Authors

KMMAU is a Korean audio understanding benchmark constructed from naturally occurring Korean audio. It is part of a Korean speech benchmark suite for evaluating SpeechLMs together with KVoiceBench and KOpenAudioBench. KMMAU contains 2,204 Korean audio understanding samples built from KSS, KMSAV, and the Seoul Corpus rather than transferred English items. It evaluates both acoustic and contextual audio understanding capabilities using multiple-choice questions grounded in Korean audio. KMMAU is constructed from target-language ASR corpora using four construction methods selected by capability: 1. Rule-based generation from speaker metadata for acoustic attributes such as age, gender, and number of speakers. 2. Rule-based generation from transcriptions for lexical questions such as word order and word frequency. 3. LLM-generated questions with human review for semantic questions such as fact extraction and topic summary. 4. Fully manual annotation for holistic capabilities that require listening to the audio, such as general counting and role/profession. The benchmark contains 646 acoustic samples and 1,558 contextual samples. All KMMAU samples are multiple-choice: gender uses 2 choices, age uses 3 choices, and the remaining capabilities use 4 choices.

提供机构:
anonyemnlpauthor18
搜集汇总
数据集介绍
anonyemnlpauthor18/KMMAU 数据集图片
构建方式
在语音语言模型评测需求日益增长的背景下,KMMAU立足于韩语原生音频资源,从KSS、KMSAV及首尔语料库中甄选素材,而非直接迁移英语基准项目,从而构建出面向韩语语音理解能力的评测数据集。其构建策略依据所需考察的能力维度分层设计:针对年龄、性别、说话人数目等声学属性,采用基于说话人元数据的规则化生成方式;针对词序、词频等词汇层面的问题,则依据转写文本进行规则化生成;对于事实抽取与主题摘要等语义理解任务,借助大语言模型生成问题并辅以人工审核;而涉及整体感知的计数与角色职业识别等任务,则完全依赖人工标注完成。
特点
该数据集共包含2204条韩语语音理解样本,涵盖九项能力子集,兼具声学与语境两个层面的考察维度。其中声学类样本646条,语境类样本1558条,全部以多项选择题形式呈现——性别子集设置2个选项,年龄子集设置3个选项,其余能力子集均采用4个选项。每条样本均配备唯一标识符、韩语语音音频、问题文本、参考答案、来源数据集名称、源样本标识以及可选的音频片段起止时间戳,字段设计完整且规范,便于研究者进行细粒度的分析与复现。
使用方法
研究者可通过Hugging Face的datasets库便捷地加载该数据集。调用load_dataset函数并指定子集名称与测试划分,即可获取对应能力的样本数据,例如加载age子集可得到276条测试样本。若需全面评估模型在韩语语音理解方面的综合表现,可依次遍历age、fact_extraction、gender、general_counting、number_of_speakers、role_profession、topic_summary、word_frequency_counting及word_order九个子集,逐一加载并统计样本规模,从而在多项选择问答框架下系统衡量语音语言模型的声学感知与语境理解能力。
背景与挑战
背景概述
随着语音语言模型(SpeechLM)在多模态理解领域的迅速崛起,面向非英语语种的音频理解评估资源却长期匮乏,韩语尤为显著。为填补这一空白,研究者于2026年构建了KMMAU基准,该数据集隶属于韩语语音基准套件,与KVoiceBench和KOpenAudioBench协同工作,旨在系统评估SpeechLM的声学与语境理解能力。KMMAU基于KSS、KMSAV及首尔语料库等本土韩语资源,而非英语数据的翻译迁移,共包含2204条多项选择样本,覆盖年龄、性别、说话人数量、词序、词频、事实抽取、主题摘要、角色职业及一般计数九项能力,其构建融合了规则生成、大模型辅助人工审核与全人工标注等多种策略。
当前挑战
韩语音频理解基准的构建面临多重挑战。从领域问题看,SpeechLM需在声学层面精准辨识年龄、性别与说话人数量,在语境层面完成词序验证、词频统计与事实抽取,而韩语丰富的形态变化与助词系统使这些任务较英语更具复杂性。就构建过程而言,确保样本均源自自然发生的韩语语音而非翻译数据,要求对源语料进行严格筛选与对齐;语义类问题依赖大模型生成后的人工审核,以保障答案的准确性与文化适切性;全人工标注的感知类任务则需投入大量专业听审精力。此外,各子集样本量差异悬殊,如词序任务达1039条而一般计数仅41条,如何维持评测的均衡性与统计效力亦构成一项持续挑战。
常用场景
经典使用场景
在语音语言模型评估领域,KMMAU作为韩语音频理解基准,其最经典的使用场景在于对SpeechLM进行多维度能力测评。研究者以该数据集中的九个能力子集为测试床,要求模型在聆听韩语自然语音后回答多项选择题,涵盖声学属性推断与语义内容理解两大范畴。此类评测范式有别于传统自动语音识别仅关注转写准确率的局限,转而审视模型对音频的整体认知水平。
解决学术问题
该数据集直面韩语语音理解评测资源匮乏这一学术痛点,弥补了以往基准多由英语题目翻译迁移而来的缺陷。通过从KSS、KMSAV及首尔语料库等本土语料中构建样本,KMMAU为探究模型在低资源语言上的声学感知与语境推理能力提供了可靠工具。其按能力分层的设计使研究者得以系统诊断模型在特定理解维度上的薄弱环节,推动了多语言语音理解研究的均衡发展。
衍生相关工作
KMMAU作为韩语语音基准套件的组成部分,与KVoiceBench及KOpenAudioBench共同构成了面向SpeechLM的评测体系。围绕该数据集衍生的经典工作包括基于规则与LLM辅助构建的韩语语音理解基准方法论,以及针对声学与语境能力分离评估的分析框架。这些工作为后续韩语语音多模态研究奠定了数据基础,并启发了对其他低资源语言类似基准的构建尝试。
以上内容由遇见数据集搜集并总结生成
二维码
社区交流群
二维码
科研交流群
商业服务