ANALOGICAL

ANALOGICAL是一个用于评估大型语言模型在长文本类比评估中的新基准。该数据集包含13个子数据集，涵盖从单词到句子的六个不同复杂度级别的类比，包括单词、单词与句子、句法、否定、蕴涵和隐喻。这些数据集用于评估八种大型语言模型在语义向量空间中识别类比对的能力。通过这些数据集，研究者发现随着类比复杂度的增加，大型语言模型识别类比的能力逐渐下降。

ANALOGICAL is a novel benchmark for evaluating large language models in long-text analogical assessment. This dataset comprises 13 sub-datasets, covering analogies across six distinct complexity levels ranging from individual words to complete sentences, including word-level, word-sentence, syntactic, negation, entailment, and metaphorical analogies. These sub-datasets are used to evaluate the capability of eight large language models to identify analogy pairs within semantic vector spaces. Through this benchmark, researchers have found that the ability of large language models to recognize analogies gradually declines as the complexity of the analogies increases.

搜集汇总

背景与挑战

背景概述

ANALOGICAL是一个用于评估大型语言模型在六个不同复杂度级别（从单词到隐喻）的类比任务中表现的数据集，包含13个子数据集。研究发现模型性能随任务复杂度增加而下降。