cobie_ai2_arc

# Dataset Card for cobie_ai2_arc

This dataset is a modification of the original [ARC](https://huggingface.co/datasets/allenai/ai2_arc) dataset for LLM cognitive bias evaluation.

## Language(s)

- English (`en`)

## Dataset Summary

ARC is a dataset of 7,787 genuine grade-school level, multiple-choice science questions, assembled to encourage research in advanced question-answering.
The dataset is partitioned into an Easy and Challenge sets, where the former contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm.

## Dataset Structure

The modifications carried out in the dataset are thought to evaluate cognitive biases in a zero-shot setting and with two different task complexities.
We only consider that have 4 multiple-choice options in the original dataset. From each original example, we create 4 different instances, each time changing the position of the correct answer (`A`, `B`, `C` or `D`).
To reduce the original task complexity, we narrow the number of choices from 4 to 3 by discarding one incorrect option at random. In this simpler variant, each example is also instanced 3 times, varying the position of the correct answer (`A`, `B` or `C`).

**Dataset Fields**

- `id`: instance id, in the format `<original_id>_<answerKey>`.
- `question`: original question.
- `choices`: a `dict` containing:
- `text`: a list of possible answers. There can be either 3 or 4 answers, depending on the task complexity.
- `label`: a list of the corresponding labels for the possible answers (`A`, `B`, `C`, `D`).
- `answerKey`: correct label.

## Additional Information

**Dataset Curators**

Language Technologies Unit (LangTech) at the Barcelona Supercomputing Center.

This work has been promoted and financed by the Generalitat de Catalunya through the [Aina](https://projecteaina.cat/) project.
This work is also funded by the Ministerio para la Transformación Digital y de la Función Pública and Plan de Recuperación, Transformación y Resiliencia - Funded by EU – NextGenerationEU within the framework of the project Desarrollo Modelos ALIA.

**License Information**

CC-BY-SA (same as [original](https://huggingface.co/datasets/allenai/ai2_arc)).

## Citation Information

```
@inproceedings{cobie,
title={Cognitive Biases, Task Complexity, and Result Intepretability in Large Language Models},
author={Mario Mina and Valle Ruiz-Fernández and Júlia Falcão and Luis Vasquez-Reina and Aitor Gonzalez-Agirre},
booktitle={Proceedings of The 31st International Conference on Computational Linguistics (COLING)},
year={2025 (to appear)}
}
```

# 数据集卡片：cobie_ai2_arc

本数据集是针对大语言模型（Large Language Model, LLM）认知偏差评估，对原始[ARC](https://huggingface.co/datasets/allenai/ai2_arc)数据集的修改版本。

## 语言
- 英语（en）

## 数据集概述
ARC是一个包含7787道真实小学阶段多项选择科学问题的数据集，旨在推动高级问答领域的研究。该数据集被划分为简单集与挑战集，其中简单集仅包含基于检索的算法与词共现算法均答错的问题。

## 数据集结构
本数据集所做的修改旨在于零样本（Zero-shot）设置下，结合两种不同的任务复杂度，评估认知偏差。我们仅保留原始数据集中拥有4个选项的样本。从每个原始样本出发，我们会生成4个不同的实例，每次调整正确答案的位置（A、B、C或D）。
为降低原始任务的复杂度，我们会通过随机丢弃一个错误选项，将选项数量从4个缩减至3个。在该简化变体中，每个样本同样会生成3个实例，调整正确答案的位置（A、B或C）。

### 数据集字段
- `id`: 实例ID，格式为`<original_id>_<answerKey>`。
- `question`: 原始问题文本。
- `choices`: 一个包含以下内容的字典：
- `text`: 候选答案列表，根据任务复杂度不同，候选答案数量可为3或4个。
- `label`: 候选答案对应的标签列表（A、B、C、D）。
- `answerKey`: 正确答案标签。

## 补充信息
### 数据集制作方
巴塞罗那超级计算中心语言技术部门（Language Technologies Unit, LangTech）。
本工作由加泰罗尼亚政府通过[Aina项目](https://projecteaina.cat/)推动并资助，同时也得到了西班牙数字化转型与公共职能部以及「复苏、转型与韧性计划」的资助——该计划由欧盟下一代欧盟（NextGenerationEU）出资，用于支持ALIA模型开发项目。

### 许可信息
CC-BY-SA许可（与[原始数据集](https://huggingface.co/datasets/allenai/ai2_arc)一致）。

## 引用信息

@inproceedings{cobie,
title={大语言模型中的认知偏差、任务复杂度与结果可解释性},
author={Mario Mina and Valle Ruiz-Fernández and Júlia Falcão and Luis Vasquez-Reina and Aitor Gonzalez-Agirre},
booktitle={第31届国际计算语言学大会（COLING）论文集},
year={2025（待刊）}
}