orionweller/NevIR
收藏资源简介：
该数据集来自论文《NevIR: Negation in Neural Information Retrieval》，旨在研究否定在神经信息检索中的影响。尽管语言模型（LMs）已成为现代信息检索（IR）架构的核心，但关于否定如何影响神经IR的研究却很少。因此，该研究构建了一个简单的基准测试，要求IR模型对仅通过否定区分的两个文档进行排序。研究结果表明，不同类型的IR架构在处理否定时的表现差异很大：交叉编码器表现最佳，其次是后期交互模型，而双编码器和稀疏神经架构表现最差。大多数当前的信息检索模型在处理否定时表现不佳，甚至比随机排序还要差。尽管通过在包含否定的对比文档数据集上继续微调可以提高性能（模型大小也有帮助），但机器与人类的表现之间仍存在较大差距。
This dataset is sourced from the paper *NevIR: Negation in Neural Information Retrieval*, which aims to investigate the impact of negation on neural information retrieval (IR). While language models (LMs) have become the core of modern information retrieval (IR) architectures, there has been relatively limited research on how negation affects neural IR. To fill this research gap, this study constructs a simple benchmark that requires IR models to rank two documents that are distinguished exclusively by negation. The study's findings demonstrate that different categories of IR architectures exhibit substantial performance disparities when handling negation: cross-encoders achieve the best performance, followed by late-interaction models, while dual encoders and sparse neural architectures perform the worst. Most current information retrieval models perform poorly when processing negation, even worse than random ranking. Although performance can be enhanced via continued fine-tuning on contrastive document datasets containing negation (model size also contributes positively to performance), a considerable gap still exists between machine and human performance.
数据集概述
- 数据集名称: NevIR
- 数据集描述: 该数据集专注于神经信息检索（Neural Information Retrieval, IR）中的否定现象研究。数据集包含了一系列文档对，这些文档对仅在否定表达上有所不同，用于评估和改进信息检索模型在处理否定情况下的性能。
数据集详细信息
- 语言: 英语
- 许可证: MIT
- 数据集大小: 1K<n<10K
- 标签: 否定、信息检索、IR
数据集结构
数据实例
- 示例: json { "id": "1-2", "WorkerId": 0, "q1": "Which mayor did more vetoing than anticipated?", "q2": "Which mayor did less vetoing than anticipated?", "doc1": "...", "doc2": "..." }
数据字段
id: 文档对唯一ID。WorkerId: 创建查询的工作者ID。q1: 仅与doc1相关的查询。q2: 仅与doc2相关的查询。doc1: 来自CondaQA的原始文档。doc2: 来自CondaQA的编辑文档。
数据分割
- 加载方式: python from datasets import load_dataset train_set = load_dataset("orionweller/nevir", "train") dev_set = load_dataset("orionweller/nevir", "validation") test_set = load_dataset("orionweller/nevir", "test")
支持的任务和排行榜
- 任务: 正确排名每对查询中的相关文档。
- 排行榜: 无官方排行榜。
引用信息
-
主要论文: bibtex @inproceedings{weller-et-al-2023-nevir, title={NevIR: Negation in Neural Information Retrieval}, author={Weller, Orion and Lawrie, Dawn, and Van Durme, Benjamin}, year={2023}, eprint={2305.07614}, archivePrefix={arXiv}, year={2023} }
-
创建初始文档的工作: bibtex @inproceedings{ravichander-et-al-2022-condaqa, title={CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation}, author={Ravichander, Abhilasha and Gardner, Matt and Marasovi{c}, Ana}, proceedings={EMNLP 2022}, year={2022} }
