seongbo/kodialogbench

--- language: - ko license: cc-by-nc-sa-4.0 tags: - dialogue - conversation - evaluation annotations_creators: - found - machine-generated pretty_name: KoDialogBench size_categories: - 10K<n<100K source_datasets: - daily_dialog - empathetic_dialogues - bavard/personachat_truecased - socialdial - aihub/k-sns - aihub/k-tdd - aihub/k-ed - aihub/k-ds task_categories: - multiple-choice configs: - config_name: dc_topic_k-sns data_files: - split: test path: dialogue_comprehension/topic/k-sns/test.jsonl - config_name: dc_topic_k-tdd data_files: - split: test path: dialogue_comprehension/topic/k-tdd/test.jsonl - config_name: dc_topic_socialdial data_files: - split: test path: dialogue_comprehension/topic/socialdial/test.jsonl - config_name: dc_emotion_k-ed data_files: - split: test path: dialogue_comprehension/emotion/k-ed/test.jsonl - config_name: dc_emotion_dailydialog data_files: - split: test path: dialogue_comprehension/emotion/dailydialog/test.jsonl - config_name: dc_emotion_empathetic data_files: - split: test path: dialogue_comprehension/emotion/empathetic_dialogues/test.jsonl - config_name: dc_relation_socialdial-distance data_files: - split: test path: dialogue_comprehension/relation/socialdial_distance/test.jsonl - config_name: dc_relation_socialdial-relation data_files: - split: test path: dialogue_comprehension/relation/socialdial_relation/test.jsonl - config_name: dc_location_socialdial data_files: - split: test path: dialogue_comprehension/location/socialdial/test.jsonl - config_name: dc_dialog_act_k-tdd data_files: - split: test path: dialogue_comprehension/dialog_act/k-tdd/test.jsonl - config_name: dc_dialog_act_dailydialog data_files: - split: test path: dialogue_comprehension/dialog_act/dailydialog/test.jsonl - config_name: dc_fact_k-ds data_files: - split: test path: dialogue_comprehension/fact/k-ds/test.jsonl - config_name: dc_fact_personachat data_files: - split: test path: dialogue_comprehension/fact/personachat/test.jsonl - config_name: dc_fact_empathetic data_files: - split: test path: dialogue_comprehension/fact/empathetic_dialogues/test.jsonl - config_name: rs_k-sns data_files: - split: test path: response_selection/k-sns/test.jsonl - config_name: rs_k-tdd data_files: - split: test path: response_selection/k-tdd/test.jsonl - config_name: rs_k-ed data_files: - split: test path: response_selection/k-ed/test.jsonl - config_name: rs_personachat data_files: - split: test path: response_selection/personachat/test.jsonl - config_name: rs_dailydialog data_files: - split: test path: response_selection/dailydialog/test.jsonl - config_name: rs_empathetic data_files: - split: test path: response_selection/empathetic_dialogues/test.jsonl - config_name: rs_socialdial data_files: - split: test path: response_selection/socialdial/test.jsonl --- ⚠️NOTE: We can't release the datasets originated from AI Hub (K-SNS, K-TDD, K-ED, K-DS) for now, according to the [Terms of Use](https://www.aihub.or.kr/intrcn/guid/usagepolicy.do?currMenu=151&topMenu=105). We're in consultation with the relevant organizations and will make these public in the appropriate form soon. # Dataset Card for KoDialogBench For most of detailed information, please refer to the following: - **Paper:** [KoDialogBench: Evaluating Conversational Understanding of Language Models with Korean Dialogue Benchmark](https://arxiv.org/abs/2402.17377) - **Repository:** [GitHub](https://github.com/sb-jang/kodialogbench) ## Dataset Details ### Dataset Description KoDialogBench is a benchmark designed to assess the conversational capabilities of language models in Korean language. To this end, we collected native Korean dialogues on daily topics from public sources (e.g., AI Hub), or translated dialogues from other languages such as English and Chinese. We then structured these conversations into diverse test datasets, spanning from dialogue comprehension to response selection tasks. This benchmark consists of 21 test sets, encompassing various aspects of open-domain colloquial dialogues (e.g., topic, emotion, dialog act). ### Data Sources We collected native Korean dialogues from AI Hub: - [K-SNS](https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=&topMenu=&aihubDataSe=data&dataSetSn=114) stands for Korean SNS (한국어 SNS) - [K-TDD](https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=&topMenu=&aihubDataSe=data&dataSetSn=543) stands for Thematic Daily Dialogues (주제별 텍스트 일상 대화 데이터) - [K-ED](https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=&topMenu=&aihubDataSe=data&dataSetSn=86) stands for Emotional Dialogues (감성 대화 말뭉치) - [K-DS](https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=&topMenu=&aihubDataSe=data&dataSetSn=117) stands for Dialogue Summary (한국어 대화 요약) We translated public datasets from other languages: - [DailyDialog](https://huggingface.co/datasets/daily_dialog) from "[DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://aclanthology.org/I17-1099/)" - [Empathetic Dialogues](https://huggingface.co/datasets/empathetic_dialogues) from "[Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset](https://aclanthology.org/P19-1534/)" - [PersonaChat](https://huggingface.co/datasets/bavard/personachat_truecased) from "[Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://aclanthology.org/P18-1205/)" - [SocialDial](https://github.com/zhanhl316/SocialDial/blob/main/human_dialogue_data.json) from "[SocialDial: A Benchmark for Socially-Aware Dialogue Systems](https://dl.acm.org/doi/10.1145/3539618.3591877)" ### Data Creation We utilized diverse meta information such as dialogue topic and speaker's emotion which was already annotated in the original datasets to formulate dialogue-related multiple-choice questions. To prevent label imbalance, we sampled the equal number of examples from each class. ### Statistics The dataset has 82,962 examples in total. | Task | Subtask | Source | # Options | # Examples | |------------------------|---------------------------|-----------------------|-----------|------------| | Dialogue Comprehension | Topic Classification | K-SNS | 6 | 1200 | | Dialogue Comprehension | Topic Classification | K-TDD | 19 | 1900 | | Dialogue Comprehension | Topic Classification | SocialDial | 4 | 400 | | Dialogue Comprehension | Emotion Recognition | K-ED | 6 | 1200 | | Dialogue Comprehension | Emotion Recognition | DailyDialog | 5 | 470 | | Dialogue Comprehension | Emotion Recognition | Empathetic Dialogues | 2 | 2000 | | Dialogue Comprehension | Relation Classification | SocialDial (Distance) | 4 | 524 | | Dialogue Comprehension | Relation Classification | SocialDial (Relation) | 3 | 330 | | Dialogue Comprehension | Location Classification | SocialDial | 4 | 376 | | Dialogue Comprehension | Dialog Act Classification | K-TDD | 4 | 520 | | Dialogue Comprehension | Dialog Act Classification | DailyDialog | 4 | 1000 | | Dialogue Comprehension | Fact Identification | K-DS | 4 | 1200 | | Dialogue Comprehension | Fact Identification | PersonaChat | 4 | 1000 | | Dialogue Comprehension | Fact Identification | Empathetic Dialogues | 4 | 2394 | | Response Selection | | K-SNS | 5 | 10295 | | Response Selection | | K-TDD | 5 | 10616 | | Response Selection | | K-ED | 5 | 17818 | | Response Selection | | PersonaChat | 5 | 7801 | | Response Selection | | DailyDialog | 5 | 6740 | | Response Selection | | Empathetic Dialogues | 5 | 7941 | | Response Selection | | SocialDial | 5 | 7237 | ## Limitations Our benchmark may suffer from a chronic problem of benchmark contamination. Due to the scarcity of Korean language resources, there is a possibility that the held-out sources utilized to construct the benchmark might overlap with training data used for some language models. ## Ethics Statement Our benchmark dataset is designed to assess capabilities related to various situations and aspects of conversations in Korean language. To achieve this, we utilized conversational content from publicly available datasets from various sources, either without modification or with translation if necessary. During this process, there is a possibility that harmful content or inappropriate biases existing in the original data may have been conveyed, or may have arisen due to limitations of translation tools. We reject any form of violence, discrimination, or offensive language, and our benchmark dataset and experimental results does not represent such values. If any harmful content or privacy infringement is identified within the dataset, we kindly request immediate notification to the authors. In the event of such cases being reported, we will apply the highest ethical standards and take appropriate actions. ## Citation **BibTeX:** ```bibtex @misc{jang2024kodialogbench, title={KoDialogBench: Evaluating Conversational Understanding of Language Models with Korean Dialogue Benchmark}, author={Seongbo Jang and Seonghyeon Lee and Hwanjo Yu}, year={2024}, eprint={2402.17377}, archivePrefix={arXiv}, primaryClass={cs.CL} } ``` ## Point of Contact [Seongbo Jang](mailto:[e-mail])

原始信息汇总

数据集概述

数据集描述

名称： KoDialogBench

语言： 韩语

许可证： CC BY-NC-SA 4.0

标签： 对话, 会话, 评估

创建者： 发现, 机器生成

数据集大小： 10K<n<100K

任务类别： 多项选择

数据集描述： KoDialogBench 是一个用于评估韩语语言模型对话能力的基准测试。该基准测试收集了来自公共来源的韩语日常对话，或将其他语言的对话翻译成韩语，并将其结构化为多样化的测试数据集，涵盖从对话理解到响应选择任务。该基准测试包含21个测试集，涵盖开放领域口语对话的各个方面（如主题、情感、对话行为）。

数据来源

原生韩语对话来源：

• K-SNS (韩国社交网络)
• K-TDD (主题日常对话)
• K-ED (情感对话)
• K-DS (对话摘要)

翻译数据来源：

• DailyDialog
• Empathetic Dialogues
• PersonaChat
• SocialDial

数据创建

利用原始数据集中已有的对话主题和说话者情感等元信息，构建了与对话相关的多项选择题。为防止标签不平衡，从每个类别中抽取了相同数量的样本。

统计信息

总样本数： 82,962

任务统计：

任务	子任务	来源	选项数	样本数
对话理解	主题分类	K-SNS	6	1200
对话理解	主题分类	K-TDD	19	1900
对话理解	主题分类	SocialDial	4	400
对话理解	情感识别	K-ED	6	1200
对话理解	情感识别	DailyDialog	5	470
对话理解	情感识别	Empathetic Dialogues	2	2000
对话理解	关系分类	SocialDial (距离)	4	524
对话理解	关系分类	SocialDial (关系)	3	330
对话理解	位置分类	SocialDial	4	376
对话理解	对话行为分类	K-TDD	4	520
对话理解	对话行为分类	DailyDialog	4	1000
对话理解	事实识别	K-DS	4	1200
对话理解	事实识别	PersonaChat	4	1000
对话理解	事实识别	Empathetic Dialogues	4	2394
响应选择		K-SNS	5	10295
响应选择		K-TDD	5	10616
响应选择		K-ED	5	17818
响应选择		PersonaChat	5	7801
响应选择		DailyDialog	5	6740
响应选择		Empathetic Dialogues	5	7941
响应选择		SocialDial	5	7237

限制

该基准测试可能存在基准污染的长期问题。由于韩语资源的稀缺性，用于构建基准测试的保留源可能与某些语言模型的训练数据重叠。

伦理声明

该基准数据集旨在评估与韩语对话中各种情境和方面相关的能力。为实现这一目标，我们利用了来自各种来源的公开可用数据集，必要时进行翻译。在此过程中，原始数据中可能存在的有害内容或不当偏见可能会被传递，或由于翻译工具的限制而产生。我们反对任何形式的暴力、歧视或攻击性语言，我们的基准数据集和实验结果不代表这些价值观。如果数据集中发现任何有害内容或隐私侵犯，请立即通知作者。在报告此类情况时，我们将采取最高伦理标准并采取适当行动。

引用

BibTeX:

bibtex @misc{jang2024kodialogbench, title={KoDialogBench: Evaluating Conversational Understanding of Language Models with Korean Dialogue Benchmark}, author={Seongbo Jang and Seonghyeon Lee and Hwanjo Yu}, year={2024}, eprint={2402.17377}, archivePrefix={arXiv}, primaryClass={cs.CL} }

联系人

Seongbo Jang

搜集汇总

数据集介绍

构建方式

KoDialogBench的构建融合了多源异构对话数据，包括从AI Hub获取的韩语原生对话（如K-SNS、K-TDD、K-ED、K-DS）以及从DailyDialog、Empathetic Dialogues、PersonaChat和SocialDial等公开数据集翻译而来的多语言对话。在数据创建过程中，研究者充分利用原始数据中已标注的元信息（如对话主题、说话者情感等），针对对话理解与响应选择两大任务，设计了多样化的多项选择题。为规避标签不平衡问题，每个类别均采样了等量样本，从而保证了评估的公正性与代表性。

特点

该数据集涵盖21个测试子集，横跨对话理解（包括主题分类、情感识别、关系分类、位置分类、对话行为分类和事实识别）与响应选择两大核心任务，全面评估模型在开放域韩语对话中的理解与生成能力。其突出特点在于多维度、多层次的任务设计，融合了原生韩语与翻译数据，且每个子集均保持类别平衡。数据规模总计超过82,000条样本，选项数量从2到19不等，为韩语对话系统的鲁棒性评估提供了系统化基准。

使用方法

KoDialogBench以HuggingFace数据集形式发布，支持通过`load_dataset`函数便捷加载。用户可根据具体任务需求，通过指定`config_name`参数选择相应的子集，例如使用`dc_topic_k-sns`进行主题分类评估，或使用`rs_dailydialog`进行响应选择测试。每个子集均提供测试集，数据格式为JSONL，可直接用于模型推理与性能评估。研究者可结合论文中的实验设置，将数据集作为韩语对话理解能力的标准化评估工具，亦可扩展至多任务学习或跨语言迁移研究。

背景与挑战

背景概述

在自然语言处理领域，对话系统的评估一直是研究的热点与难点，尤其是面向非英语语言如韩语时，缺乏高质量、多维度的评测基准。KoDialogBench数据集由Seongbo Jang、Seonghyeon Lee和Hwanjo Yu于2024年提出，主要来自韩国浦项科技大学（POSTECH）的研究团队。该数据集旨在系统评估语言模型在韩语对话中的理解能力，涵盖了话题分类、情感识别、对话行为分类、事实识别以及响应选择等21项子任务。通过整合来自AI Hub的韩语原生对话数据与翻译自英语、中文的公开数据集（如DailyDialog、Empathetic Dialogues等），KoDialogBench为韩语对话理解研究提供了首个综合性基准，填补了该领域的空白，对推动多语言对话AI的发展具有重要影响力。

当前挑战

KoDialogBench面临的挑战首先体现在领域问题的复杂性上：对话理解涉及多维度语义推理，如情感、意图、关系等，现有模型难以在单一框架内统一处理这些异构任务，尤其在韩语这种形态丰富、语序灵活的语言中，细粒度语义的捕捉更为困难。其次，在数据集构建过程中，研究人员需应对跨源数据整合的挑战，包括不同数据集标注体系不一致、翻译质量波动（如SocialDial的情感标签在韩语中的文化适配问题）以及AI Hub部分数据因版权限制无法直接公开，导致基准的复现与扩展受限。此外，由于韩语公开资源稀缺，基准污染风险较高，部分测试数据可能与预训练语料重叠，影响评估的公平性。

常用场景

经典使用场景

KoDialogBench作为专为韩语设计的对话理解基准测试，其经典使用场景在于对语言模型的多维度对话能力进行系统评估。该数据集涵盖21个测试子集，巧妙融合了源自AI Hub的原生韩语语料与从英语、中文等语言精译而来的对话资源，构建了从对话理解到回复选择的完整评估框架。研究者可利用其多样化的多选问答任务，对模型在主题分类、情感识别、关系推断、对话行为分析及事实核查等核心维度上的表现进行深入剖析，从而全面衡量模型在开放域韩语对话场景中的综合理解能力。

实际应用

在实际应用层面，KoDialogBench为韩语对话系统的开发与部署提供了关键的评估与优化指导。该基准可广泛应用于韩语智能客服、情感交互机器人、社交对话代理等商业场景，帮助开发者精准识别模型在主题理解、情感共鸣、关系辨识等方面的优势与不足。通过系统化的多维度评估，企业能够更有针对性地优化对话模型，提升其在真实韩语环境中的交互质量与用户体验，从而加速韩语人工智能对话产品从研究走向落地的进程。

衍生相关工作

KoDialogBench的发布催生了一系列围绕韩语对话理解的研究工作。该基准启发了针对低资源语言对话评估方法的探索，推动了跨语言对话理解模型的迁移学习研究。同时，其多任务评估框架为开发更强大的韩语预训练语言模型提供了重要的性能衡量标准，促进了模型在情感识别、对话行为分析等子任务上的专项优化。此外，该数据集还激励了研究者关注韩语对话中特有的文化语境与社交规范，衍生出对对话系统社会感知能力评估的新方向，进一步丰富了韩语自然语言处理领域的研究生态。