E-KAR (Benchmark for Explainable Knowledge-intensive Analogical Reasoning)

识别类比的能力是人类认知的基础。测试单词类比的现有基准并未揭示神经模型类比推理的底层过程。我们坚信能够推理的模型应该是正确的，因此我们提出了首创的可解释知识密集型类比推理基准 (E-KAR)。我们的基准测试包括来自公务员考试的 1,655 个（中文）和 1,251 个（英文）问题，这些问题需要深入的背景知识才能解决。更重要的是，我们设计了一个自由文本解释方案来解释是否应该进行类比，并为每个问题和候选答案手动注释它们。

The ability to recognize analogies is fundamental to human cognition. Existing benchmarks for word analogy testing fail to reveal the underlying mechanisms of analogical reasoning in neural models. We firmly believe that a model capable of reasoning should be able to produce valid outputs, so we propose the first interpretable knowledge-intensive analogical reasoning benchmark (E-KAR). Our benchmark consists of 1,655 (Chinese) and 1,251 (English) questions sourced from civil service examinations, which demand in-depth background knowledge to solve. More importantly, we design a free-text explanation framework to clarify whether an analogy is applicable, and manually annotate each question along with its candidate answers.

搜集汇总

背景与挑战

背景概述

E-KAR是一个用于可解释知识密集型类比推理的基准数据集，包含来自公务员考试的1,655个中文问题和1,251个英文问题，这些问题需要背景知识来解决，并提供了手动注释的自由文本解释方案。