CrossRE

关系提取 (RE) 引起了越来越多的关注，但是当前的RE评估仅限于域内评估设置。关于RE系统在具有挑战性但现实的发行外评估设置中的表现如何，鲜为人知。为了解决这一差距，我们提出了CrossRE，这是一种新的，免费的RE跨域基准，它包括六个不同的文本域，并包括多标签注释。另一项创新是，我们发布注释期间收集的元数据，以包括困难实例的解释和标志。我们通过最先进的关系分类模型提供经验评估。由于元数据使我们能够对最新模型进行新的了解，因此我们对困难案例的影响进行了全面分析，并找到了模型与人类注释之间的相关性。总体而言，我们的实证研究突出了跨域RE的难度。我们发布我们的数据集，以刺激更多的研究。

Relation Extraction (RE) has attracted growing attention, but current RE evaluations are limited to the in-domain evaluation setting. Little is known about how RE systems perform in the challenging yet realistic out-of-domain evaluation setting. To address this gap, we propose CrossRE, a novel, freely available cross-domain benchmark for RE that encompasses six distinct text domains and includes multi-label annotations. Another innovation is that we release the metadata collected during the annotation process, which includes explanations and flags for difficult instances. We conduct empirical evaluations using state-of-the-art relation classification models. As the metadata allows us to gain new insights into state-of-the-art models, we perform a comprehensive analysis of the impact of hard cases and uncover correlations between model behaviors and human annotations. Overall, our empirical study highlights the challenges of cross-domain RE. We release our dataset to encourage further research in this field.

搜集汇总

背景与挑战

背景概述

CrossRE是一个用于关系提取的跨域基准数据集，包含六个不同文本域和多标签注释，旨在评估RE系统在域外设置中的性能。该数据集创新性地提供了注释期间的元数据，以支持对模型困难案例的分析，并促进跨域RE研究的发展。