sapienzanlp/MMLU-Adversarial

MMLU-Adversarial是一个用于诊断的数据集，旨在评估当前基于LLM的答案提取技术在检测模型因虚构或错误的推理而产生无效答案的能力。数据集中的每个实例都包含一个破坏最终选定答案有效性的推理链，并应标记为无效（例如[No Valid Answer]）。错误的推理分为两类：不连贯推理和多个答案。不连贯推理是指支持一个答案的推理路径，但结论却是另一个答案，没有为这种转变提供连贯的合理性。多个答案是指同时支持多个答案的推理路径，导致模糊或内部矛盾。

MMLU-Adversarial is a diagnostic dataset designed to evaluate the ability of current LLM-based answer extraction techniques to detect instances in which the model produces invalid answers due to hallucinated or flawed reasoning. Each instance in the dataset includes a reasoning chain that undermines the validity of the final selected answer, and as such, should be labeled as invalid (e.g., [No Valid Answer]). The flawed reasoning falls into one of two categories: Inconsistent reasoning, where a reasoning path supports one answer but concludes with a different one without providing a coherent justification for the shift; and Multiple answers, where a reasoning path simultaneously supports more than one answer, leading to ambiguity or internal contradiction.