kanishka/comps

COMPS是一个包含英语最小对句的数据集，用于测试语言模型对概念及其属性的知识。特别地，它测试了语言模型将属性归因于日常概念的能力，并展示与属性继承兼容的推理，其中从属概念继承其上级概念（上位词）的属性。

原始信息汇总

数据集卡片 for "COMPS"

数据集描述

COMPS 是一个英语中的最小对偶句数据集，用于测试语言模型（LMs）对概念及其属性的知识。具体来说，它测试 LMs 将属性归因于日常概念的能力，并展示与属性继承兼容的推理，其中下属概念继承其上级（超词）的属性。

基本信息

• 注释创建者: 专家生成
• 语言创建者: 机器生成
• 语言: 英语
• 许可证: Apache 2.0
• 多语言性: 单语
• 大小类别: 10K<n<100K
• 源数据集: 原始数据集

引用信息

@inproceedings{misra-etal-2023-comps, title = "{COMPS}: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Models", author = "Misra, Kanishka and Rayz, Julia and Ettinger, Allyson", booktitle = "Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics", month = may, year = "2023", address = "Dubrovnik, Croatia", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2023.eacl-main.213", doi = "10.18653/v1/2023.eacl-main.213", pages = "2928--2949", abstract = "A characteristic feature of human semantic cognition is its ability to not only store and retrieve the properties of concepts observed through experience, but to also facilitate the inheritance of properties (can breathe) from superordinate concepts (animal) to their subordinates (dog){---}i.e. demonstrate property inheritance. In this paper, we present COMPS, a collection of minimal pair sentences that jointly tests pre-trained language models (PLMs) on their ability to attribute properties to concepts and their ability to demonstrate property inheritance behavior. Analyses of 22 different PLMs on COMPS reveal that they can easily distinguish between concepts on the basis of a property when they are trivially different, but find it relatively difficult when concepts are related on the basis of nuanced knowledge representations. Furthermore, we find that PLMs can show behaviors suggesting successful property inheritance in simple contexts, but fail in the presence of distracting information, which decreases the performance of many models sometimes even below chance. This lack of robustness in demonstrating simple reasoning raises important questions about PLMs{} capacity to make correct inferences even when they appear to possess the prerequisite knowledge.", }