BSC-LT/cobie_ai2_arc

该数据集是对原始ARC数据集的修改，用于评估大语言模型的认知偏差。ARC数据集包含7,787个小学水平的多项选择科学问题，旨在促进高级问答系统的研究。数据集分为简单和挑战两个部分，其中简单部分仅包含检索算法和词共现算法都未能正确回答的问题。在修改后的数据集中，每个原始问题被实例化为4个不同的实例，每次改变正确答案的位置（A、B、C或D）。为了降低任务复杂度，还将选项数量从4个减少到3个，随机丢弃一个错误选项，并在简化版本中每个问题实例化为3次，改变正确答案的位置（A、B或C）。

The ARC dataset is a collection of 7,787 genuine grade-school level, multiple-choice science questions, designed to encourage research in advanced question-answering. The dataset is partitioned into Easy and Challenge subsets, with the former containing only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm. The modifications to the dataset aim to evaluate cognitive biases in a zero-shot setting and with two different task complexities. Each original example is created into 4 different instances by changing the position of the correct answer. To reduce task complexity, the number of choices is narrowed from 4 to 3 by discarding one incorrect option at random, with each example also instanced 3 times, varying the position of the correct answer. The dataset fields include id, question, choices, and answerKey.