irds/beir_msmarco

`beir/msmarco`数据集由`ir-datasets`包提供，主要用于文本检索任务。数据集包含两个主要部分：`docs`（文档，即语料库）和`queries`（查询，即主题）。文档数量为8,841,823条，查询数量为509,962条。该数据集被用于`beir_msmarco_dev`、`beir_msmarco_test`和`beir_msmarco_train`等子数据集。

原始信息汇总

数据集概述

数据集名称

beir/msmarco

数据提供者

由 ir-datasets 包提供。

数据内容

• docs (文档，即语料库); 数量: 8,841,823
• queries (查询，即主题); 数量: 509,962

使用场景

该数据集被用于以下场景：

• beir_msmarco_dev
• beir_msmarco_test
• beir_msmarco_train

数据集加载示例

python from datasets import load_dataset

docs = load_dataset(irds/beir_msmarco, docs) for record in docs: record # {doc_id: ..., text: ...}

queries = load_dataset(irds/beir_msmarco, queries) for record in queries: record # {query_id: ..., text: ...}

引用信息

@inproceedings{Bajaj2016Msmarco, title={MS MARCO: A Human Generated MAchine Reading COmprehension Dataset}, author={Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, Tong Wang}, booktitle={InCoCo@NIPS}, year={2016} } @article{Thakur2021Beir, title = "BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models", author = "Thakur, Nandan and Reimers, Nils and Rücklé, Andreas and Srivastava, Abhishek and Gurevych, Iryna", journal= "arXiv preprint arXiv:2104.08663", month = "4", year = "2021", url = "https://arxiv.org/abs/2104.08663", }