plutus-QA

----------------------------------------------------------------
# Dataset Card for Plutus QA

## Table of Contents

- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks](#supported-tasks)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://huggingface.co/collections/TheFinAI/plutus-benchmarking-greek-financial-llms-67bc718fb8d897c65f1e87db
- **Repository:** https://huggingface.co/datasets/TheFinAI/plutus-QA
- **Paper:** Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance
- **Leaderboard:** https://huggingface.co/spaces/TheFinAI/Open-Greek-Financial-LLM-Leaderboard#/
- **Model:** https://huggingface.co/spaces/TheFinAI/plutus-8B-instruct

### Dataset Summary

Plutus QA is a question-answering dataset designed for financial applications in the Greek language. This resource contains a diverse set of queries, each paired with an answer, additional context text, a set of multiple-choice options, and a gold label index indicating the correct choice. By integrating textual context with a multiple-choice format, the dataset is aimed at benchmarking the capabilities of large language models in resolving financial questions in low-resource settings.

### Supported Tasks

- **Task:** Question Answering
- **Evaluation Metrics:** Accuracy

### Languages

- Greek

## Dataset Structure

### Data Instances

Each instance in the dataset consists of the following five fields:

- **query:** A question or prompt regarding financial matters.
- **answer:** The corresponding answer text paired with the query.
- **text:** Additional context or background information to support the query.
- **choices:** A sequence field containing multiple-choice answer options.
- **gold:** An integer field representing the index of the correct answer in the choices.

### Data Fields

- **query:** String – Represents the financial question or prompt.
- **answer:** String – Contains the answer aligned with the given query.
- **text:** String – Provides extra contextual information related to the query.
- **choices:** Sequence of strings – Lists all available answer options.
- **gold:** Int64 – Denotes the index of the correct answer from the choices.

### Data Splits

The dataset is organized into three splits:

- **Train:** 267 instances (474,769 bytes)
- **Validation:** 48 instances (74,374 bytes)
- **Test:** 225 instances (387,629 bytes)

## Dataset Creation

### Curation Rationale

The Plutus QA dataset was created to enable the evaluation of large language models on question-answering tasks within the financial domain for Greek language texts. Its design—including multiple-choice answers with additional context—aims to reflect complex financial decision-making and reasoning processes in a low-resource language environment.

### Source Data

#### Initial Data Collection and Normalization

The source data is derived from Greek university financial exams. Standardization and normalization procedures were applied to ensure consistency across queries, choices, and textual context.

#### Who are the Source Language Producers?

Greek university financial exams.

### Annotations

#### Annotation Process

Annotations were performed by domain experts proficient in both finance and linguistics. The process included verifying the correct answer for each query and marking the corresponding correct index among provided choices.

#### Who are the Annotators?

A team of financial analysts and linguists collaborated to ensure the annotations are accurate and reflective of real-world financial reasoning.

### Personal and Sensitive Information

This dataset is curated to exclude any personally identifiable information (PII) and only contains public financial text data necessary for question-answering tasks.

## Considerations for Using the Data

### Social Impact of Dataset

The Plutus QA dataset is instrumental in enhancing automated question-answering systems in the financial sector, particularly for Greek language applications. Improved QA systems can support better financial decision-making, increase the efficiency of financial services, and contribute to academic research in financial natural language processing.

### Discussion of Biases

- The domain-specific financial language may limit generalizability to non-financial question-answering tasks.
- Annotation subjectivity could introduce biases in determining the correct answer among multiple choices.
- The dataset's focus on Greek financial documents may not fully represent other financial or multilingual contexts.

### Other Known Limitations

- Pre-processing may be required to handle variations in question and answer formats.
- The dataset is specialized for the financial domain and may need adaptation for different QA tasks or domains.

## Additional Information

### Dataset Curators

- Xueqing Peng
- Triantafillos Papadopoulos
- Efstathia Soufleri
- Polydoros Giannouris
- Ruoyu Xiang
- Yan Wang
- Lingfei Qian
- Jimin Huang
- Qianqian Xie
- Sophia Ananiadou

The research is supported by NaCTeM, Archimedes RC, and The Fin AI.

### Licensing Information

- **License:** Apache License 2.0

### Citation Information

If you use this dataset in your research, please consider citing it as follows:

```bibtex
@misc{peng2025plutusbenchmarkinglargelanguage,
title={Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance},
author={Xueqing Peng and Triantafillos Papadopoulos and Efstathia Soufleri and Polydoros Giannouris and Ruoyu Xiang and Yan Wang and Lingfei Qian and Jimin Huang and Qianqian Xie and Sophia Ananiadou},
year={2025},
eprint={2502.18772},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2502.18772},
}
```

# Plutus QA 数据集卡片

## 目录
- [目录](#table-of-contents)
- [数据集描述](#dataset-description)
- [数据集摘要](#dataset-summary)
- [支持任务](#supported-tasks)
- [语言](#languages)
- [数据集结构](#dataset-structure)
- [数据实例](#data-instances)
- [数据字段](#data-fields)
- [数据划分](#data-splits)
- [数据集构建](#dataset-creation)
- [数据遴选依据](#curation-rationale)
- [源数据](#source-data)
- [标注](#annotations)
- [个人与敏感信息](#personal-and-sensitive-information)
- [数据集使用注意事项](#considerations-for-using-the-data)
- [数据集的社会影响](#social-impact-of-dataset)
- [偏差讨论](#discussion-of-biases)
- [其他已知局限性](#other-known-limitations)
- [附加信息](#additional-information)
- [数据集编撰者](#dataset-curators)
- [许可信息](#licensing-information)
- [引用信息](#citation-information)
- [贡献](#contributions)

## 数据集描述

- **主页：** https://huggingface.co/collections/TheFinAI/plutus-benchmarking-greek-financial-llms-67bc718fb8d897c65f1e87db
- **仓库：** https://huggingface.co/datasets/TheFinAI/plutus-QA
- **论文：** Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance
- **排行榜：** https://huggingface.co/spaces/TheFinAI/Open-Greek-Financial-LLM-Leaderboard#/
- **模型：** https://huggingface.co/spaces/TheFinAI/plutus-8B-instruct

### 数据集摘要

Plutus QA 是一款面向希腊语金融应用的问答（Question Answering）数据集。该资源包含多样化的查询集合，每条查询均配有对应答案、额外上下文文本、一组多项选择选项，以及指示正确选项的金标索引（gold label index）。通过将文本上下文与多项选择格式相结合，本数据集旨在针对低资源语言环境下的大语言模型（Large Language Model）金融问答能力开展基准测试。

### 支持任务

- **任务：** 问答（Question Answering）
- **评估指标：** 准确率（Accuracy）

### 语言

- **语言：** 希腊语

## 数据集结构

### 数据实例

本数据集的每条实例均包含以下五个字段：

- **query：** 与金融事项相关的问题或提示。
- **answer：** 与该查询对应的答案文本。
- **text：** 用于辅助查询的额外上下文或背景信息。
- **choices：** 包含多项选择答案的序列字段。
- **gold：** 表示choices中正确答案索引的整数字段。

### 数据字段

- **query：** 字符串类型 —— 代表金融问题或提示。
- **answer：** 字符串类型 —— 包含与给定查询匹配的答案。
- **text：** 字符串类型 —— 提供与查询相关的额外上下文信息。
- **choices：** 字符串序列 —— 列出所有可用的答案选项。
- **gold：** Int64 类型 —— 表示choices中正确答案的索引。

### 数据划分

本数据集分为三个子集：

- **训练集（Train）：** 267 条实例（474,769 字节）
- **验证集（Validation）：** 48 条实例（74,374 字节）
- **测试集（Test）：** 225 条实例（387,629 字节）

## 数据集构建

### 数据遴选依据

Plutus QA 数据集的构建旨在实现针对希腊语文本金融领域问答任务的大语言模型评估。其设计（包含带额外上下文的多项选择答案）旨在反映低资源语言环境下的复杂金融决策与推理过程。

### 源数据

#### 初始数据收集与标准化

源数据取自希腊高校金融考试。我们采用了标准化与归一化流程，以确保查询、选项与文本上下文的一致性。

#### 源语言内容生产者是谁？

希腊高校金融考试的命题方。

### 标注

#### 标注流程

标注工作由精通金融与语言学的领域专家完成。流程包括验证每条查询的正确答案，并标记对应选项中的正确索引。

#### 标注人员是谁？

由一支金融分析师与语言学家组成的团队开展标注工作，以确保标注准确且贴合真实金融推理场景。

### 个人与敏感信息

本数据集已剔除所有个人身份信息（PII），仅包含问答任务所需的公开金融文本数据。

## 数据集使用注意事项

### 数据集的社会影响

Plutus QA 数据集有助于提升金融领域的自动化问答系统性能，尤其针对希腊语应用场景。优化后的问答系统可辅助更优金融决策、提升金融服务效率，并为金融自然语言处理的学术研究提供支持。

### 偏差讨论

- 领域特定的金融语言可能限制其在非金融问答任务中的泛化能力。
- 标注主观性可能会在多项选择的正确答案判定中引入偏差。
- 数据集聚焦于希腊金融文档，可能无法完全代表其他金融或多语言场景。

### 其他已知局限性

- 可能需要进行预处理以适配查询与答案格式的变体。
- 本数据集专为金融领域定制，若要适配其他问答任务或领域则需进行调整。

## 附加信息

### 数据集编撰者

- Xueqing Peng
- Triantafillos Papadopoulos
- Efstathia Soufleri
- Polydoros Giannouris
- Ruoyu Xiang
- Yan Wang
- Lingfei Qian
- Jimin Huang
- Qianqian Xie
- Sophia Ananiadou

本研究由 NaCTeM、Archimedes RC 及 The Fin AI 资助。

### 许可信息

- **许可协议：** Apache License 2.0

### 引用信息

若您在研究中使用本数据集，请按以下格式引用：

bibtex
@misc{peng2025plutusbenchmarkinglargelanguage,
title={Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance},
author={Xueqing Peng and Triantafillos Papadopoulos and Efstathia Soufleri and Polydoros Giannouris and Ruoyu Xiang and Yan Wang and Lingfei Qian and Jimin Huang and Qianqian Xie and Sophia Ananiadou},
year={2025},
eprint={2502.18772},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2502.18772},
}

### 贡献