davidkim205/FinDartBench

---
language:
- ko
license: cc-by-nc-4.0
pretty_name: FinDartBench
size_categories:
- 10K<n<100K
task_categories:
- question-answering
tags:
- finance
- korean
- open-domain
---

# FinDartBench

FinDartBench is a Korean financial question answering benchmark built from DART disclosure filings.
It is designed to evaluate real-world financial document understanding by pairing context-grounded questions with high-quality reference answers validated through a multi-stage LLM-based pipeline.

Unlike simple synthetic QA datasets, FinDartBench emphasizes **grounding, answer quality, and inter-model consensus**, making it suitable for reliable evaluation of financial QA systems.

For a detailed description of the dataset and construction pipeline, please refer to the 📄[technical report](https://davidkim205.github.io/findartbench.html).

This work was supported by the Ministry of Science and ICT.

## Tasks

- Korean financial document question answering
- Open-book QA over corporate disclosure documents
- Answer evaluation with multiple ranked reference answers

## Dataset Overview

- **Total examples:** 14,444
- **Total reference answers:** 39,488
- **Companies:** 10 major Korean companies
- **Source documents:** ~200 DART filings
- **Language:** Korean
- **License:** CC BY-NC 4.0

## Data Fields

| key | type | description |
| -------------- | ---------- | --------------------------------------------- |
| id | int | Unique identifier for each QA instance |
| doc_id | int | Identifier for the source document |
| company | string | Source company name (Korean) |
| doc_type | string | Type of disclosure document |
| context | string | Grounding document chunk |
| question | string | Korean question derived from context |
| answers | list[dict] | Ranked reference answers |
| answers.model | string | Model used for answer generation |
| answers.answer | string | Answer text in Korean |

## Example Instance

```json
{
"id": 11011,
"doc_id": 352052,
"company": "현대자동차",
"doc_type": "주주총회소집공고",
"context": "### II. 최대주주등과의 거래내역에 관한 사항\n\n...",
"question": "현대글로비스와의 거래금액 산정 기준과 기타 거래금액 산정 기준은 어떻게 다른가?",
"answers": [
{"model": "DeepSeek-V3.2-Exp", "answer": "..."},
{"model": "Kimi-K2.5", "answer": "..."}
]
}
```

Reference answers are ordered by quality after validation.

## Data Construction Pipeline

FinDartBench is constructed through a multi-stage pipeline that ensures both **diversity** and **reliability** of QA pairs:

1. **Document Processing**

DART filings are segmented into structured chunks while preserving document hierarchy.

2. **Question Generation & Deduplication**

Multiple LLMs generate candidate questions, which are then clustered to remove duplicates and select representative questions.

3. **Answer Generation**

Multiple LLMs produce diverse candidate answers for each question.

4. **Quality Validation**

Candidate answers are filtered based on:

* grounding to the context
* Korean language quality
* inter-model agreement (consensus)

## Dataset Statistics

### Document Type Distribution

| doc_type | count |
| ------------- | ----: |
| 사업보고서 | 5,638 |
| 기업지배구조보고서공시 | 2,699 |
| 주주총회소집공고 | 1,749 |
| 투자설명서 | 1,019 |
| 의결권대리행사권유참고서류 | 552 |
| 기타 | 2,787 |

### Company Distribution

| company | LG전자 | SK텔레콤 | 삼성전자 | 현대자동차 | 한국전력 | SK하이닉스 | 국민은행 | 기아 | HMM | 두나무 |
| :-----: | ----- | ----- | ----- | ----- | ----- | ------ | ---- | --- | --- | --- |
| count | 3,924 | 2,295 | 2,036 | 1,654 | 1,429 | 1,115 | 799 | 500 | 447 | 245 |

## Source Data

All data is derived from publicly available corporate disclosures provided by the Financial Supervisory Service (DART):

[https://dart.fss.or.kr/](https://dart.fss.or.kr/)

## Limitations

* The dataset reflects structures specific to Korean disclosure documents
* Automatically generated using LLMs; residual errors may exist
* Limited coverage (10 companies, ~200 documents)

## Acknowledgements

This research was supported by the “Advanced GPU Utilization Support Program(Beta Service)” funded by the Government of the Republic of Korea (Ministry of Science and ICT).


---
语言：
- ko
许可证：CC BY-NC 4.0
规范名称：FinDartBench
规模类别：
- 1万<n<10万
任务类别：
- 问答任务
标签：
- 金融
- 韩语
- 开放域
---

# FinDartBench

FinDartBench是一款基于DART公开披露文件构建的韩语金融问答基准数据集。该数据集旨在通过将基于上下文锚定（grounding）的问题与经过多阶段大语言模型（LLM）管线验证的高质量参考答案配对，来评估模型对真实世界金融文档的理解能力。与简易合成问答数据集不同，FinDartBench着重突出**上下文锚定、答案质量与模型间共识**三大核心特性，使其适用于金融问答系统的可靠评估。

如需了解该数据集与构建管线的详细细节，请参阅📄[技术报告](https://davidkim205.github.io/findartbench.html)。本研究获韩国科学与信息通信技术部资助。

## 任务

- 韩语金融文档问答
- 企业披露文件的开卷问答
- 基于多排序参考答案的答案评估

## 数据集概览

- **总样本数：14,444条**
- **总参考答案数：39,488条**
- **覆盖企业：10家韩国头部企业**
- **源文件：约200份DART披露文件**
- **语言：韩语**
- **许可证：CC BY-NC 4.0**

## 数据字段

| 字段名 | 数据类型 | 说明 |
| -------------- | ---------- | --------------------------------------------- |
| id | 整数型 | 每个问答实例的唯一标识符 |
| doc_id | 整数型 | 源文件标识符 |
| company | 字符串型 | 源企业名称（韩语） |
| doc_type | 字符串型 | 披露文件类型 |
| context | 字符串型 | 锚定上下文的文档片段 |
| question | 字符串型 | 基于上下文生成的韩语问题 |
| answers | 字典列表型 | 按排序排列的参考答案集合 |
| answers.model | 字符串型 | 用于生成答案的大语言模型名称 |
| answers.answer | 字符串型 | 韩语格式的答案文本 |

## 示例实例

json
{
"id": 11011,
"doc_id": 352052,
"company": "현대자동차",
"doc_type": "주주총회소집공고",
"context": "### II. 최대주주등과의 거래내역에 관한 사항

...",
"question": "현대글로비스와의 거래금액 산정 기준과 기타 거래금액 산정 기준은 어떻게 다른가?",
"answers": [
{"model": "DeepSeek-V3.2-Exp", "answer": "..."},
{"model": "Kimi-K2.5", "answer": "..."}
]
}

参考答案已按验证后的质量进行排序。

## 数据构建管线

FinDartBench通过多阶段管线构建，确保问答对的**多样性**与**可靠性**：
1. **文档处理**
将DART披露文件分割为结构化片段，并保留文档层级结构。
2. **问题生成与去重**
由多个大语言模型生成候选问题，随后通过聚类去除重复项并筛选代表性问题。
3. **答案生成**
多个大语言模型为每个问题生成多样化的候选答案。
4. **质量验证**
候选答案将基于以下标准进行筛选：
* 上下文锚定合规性
* 韩语语言质量
* 模型间共识度

## 数据集统计

### 文档类型分布

| 文档类型 | 样本数量 |
| ------------- | ----: |
| 业务报告 | 5,638 |
| 企业治理结构报告公告 | 2,699 |
| 股东大会召集公告 | 1,749 |
| 投资说明书 | 1,019 |
| 表决权代理行使参考文件 | 552 |
| 其他 | 2,787 |

### 企业分布

| 企业名称 | LG电子 | SK电讯 | 三星电子 | 现代汽车 | 韩国电力 | SK海力士 | 国民银行 | 起亚 | 现代商船 | 斗南宇 |
| :---------: | ----- | ----- | ----- | ----- | ----- | ------ | ---- | --- | --- | --- |
| 样本数量 | 3,924 | 2,295 | 2,036 | 1,654 | 1,429 | 1,115 | 799 | 500 | 447 | 245 |

## 源数据

所有数据均源自韩国金融监督院提供的公开企业披露文件（DART平台）：
[https://dart.fss.or.kr/](https://dart.fss.or.kr/)

## 局限性说明
* 该数据集反映了韩国披露文件特有的结构规范
* 数据集由大语言模型自动生成，可能存在残留错误
* 覆盖范围有限（仅10家企业，约200份文件）

## 致谢
本研究获大韩民国政府（科学与信息通信技术部）资助的"高级GPU利用支持计划（测试版服务）"支持。