anonyemnlpauthor18/KMMAU

KMMAU是一个基于自然产生的韩语音频构建的韩语音频理解基准测试集。它是用于评估SpeechLMs的韩语语音基准测试套件的一部分，与KVoiceBench和KOpenAudioBench共同构成。该数据集包含从KSS、KMSAV和首尔语料库中构建的2,204个韩语音频理解样本，而非从英语项目转换而来。它通过基于韩语音频的多选题来评估声学和上下文音频理解能力。KMMAU使用四种构建方法从目标语言的ASR语料库中构建，根据能力选择方法：1. 基于说话者元数据的规则生成，用于年龄、性别和说话者数量等声学属性；2. 基于转录文本的规则生成，用于词序和词频等词汇问题；3. LLM生成问题并人工审核，用于事实提取和主题摘要等语义问题；4. 完全手动标注，用于需要听音频的整体能力，如一般计数和角色/职业识别。基准测试包含646个声学样本和1,558个上下文样本。所有KMMAU样本均为多选题：性别使用2个选项，年龄使用3个选项，其余能力使用4个选项。

KMMAU is a Korean audio understanding benchmark constructed from naturally occurring Korean audio. It is part of a Korean speech benchmark suite for evaluating SpeechLMs together with KVoiceBench and KOpenAudioBench. KMMAU contains 2,204 Korean audio understanding samples built from KSS, KMSAV, and the Seoul Corpus rather than transferred English items. It evaluates both acoustic and contextual audio understanding capabilities using multiple-choice questions grounded in Korean audio. KMMAU is constructed from target-language ASR corpora using four construction methods selected by capability: 1. Rule-based generation from speaker metadata for acoustic attributes such as age, gender, and number of speakers. 2. Rule-based generation from transcriptions for lexical questions such as word order and word frequency. 3. LLM-generated questions with human review for semantic questions such as fact extraction and topic summary. 4. Fully manual annotation for holistic capabilities that require listening to the audio, such as general counting and role/profession. The benchmark contains 646 acoustic samples and 1,558 contextual samples. All KMMAU samples are multiple-choice: gender uses 2 choices, age uses 3 choices, and the remaining capabilities use 4 choices.