orgrctera/msmarco_document_ranking

--- license: other task_categories: - text-retrieval language: - en tags: - information-retrieval - document-ranking - msmarco - benchmark - retrieval pretty_name: MS MARCO Document Ranking (CTERA format) size_categories: - "100K<n<1M" --- # MS MARCO Document Ranking ## Dataset description and background **MS MARCO** (MicroSoft MAchine Reading COmprehension) is a large-scale collection originally introduced for machine reading comprehension and question answering. Over time it has become a standard benchmark for **information retrieval** under abundant training data: hundreds of thousands of queries with human relevance signals, aligned with real web search behavior. The **document ranking** track uses a **document-level corpus** derived from the same ecosystem as MS MARCO passage ranking. The official resource describes a corpus on the order of **~3.2M documents**, with **training queries on the order of hundreds of thousands**, **development** and **leaderboard test** query sets, and **TREC-style qrels** (query–relevance judgments) for training and development. For training, **passage-level labels are mapped to document IDs** under the assumption that a document containing a judged-relevant passage is treated as a relevant document—supporting transfer between passage-focused and document-focused retrieval research. This Hugging Face dataset is a **CTERA-packaged view** of that task: each row pairs a **natural-language query** with **structured labels/metadata** suitable for retrieval and RAG benchmarking (see [Data fields](#data-fields) below). **Official MS MARCO ranking resources (corpus, qrels, leaderboards):** [MS MARCO — Datasets for Document and Passage Ranking](https://microsoft.github.io/msmarco/Datasets.html) ## Task: document ranking / retrieval This dataset supports **ad-hoc document retrieval / ranking**: given a **query**, a system should **rank documents** from a collection by **relevance**. In research settings this is often split into: - **Full ranking (retrieval):** score or retrieve from the **full document collection** (official submissions allow a bounded number of documents per query, e.g. up to **100** in the MS MARCO document ranking setup). - **Top‑k reranking:** rerank a fixed candidate list (e.g. **top‑100** candidates from a first-stage retriever)—a common production pattern of “retrieve, then rerank.” **Evaluation** in the MS MARCO ranking leaderboards is typically reported with **MRR@10** (Mean Reciprocal Rank at rank 10) for the document ranking task, alongside standard TREC-style analyses where applicable. (Consult the official leaderboard and TREC Deep Learning track materials for the exact metric definitions used in a given campaign.) ## Data fields Parquet splits expose three columns: | Column | Description | |--------|-------------| | `input` | The query text (string). | | `expected_output` | JSON string with relevance information; **format differs by split** (see examples). | | `metadata` | JSON string with identifiers and benchmark tags (`benchmark_name`, `split`, `query_id`, etc.). | **Splits:** `train`, `dev`, and `test` are provided. The **test** split may use **held-out** or **empty** labels in `expected_output` for blind evaluation workflows—check the sample below. ## Examples **Train** (`expected_output` includes query id and judged relevant document id(s) as JSON): ```json { "input": ")what was the immediate impact of the success of the manhattan project?", "expected_output": "{\"qid\": \"1185869\", \"qrels\": [\"D59219\"]}", "metadata": "{\"query_id\": \"1185869\", \"split\": \"train\", \"benchmark_name\": \"msmarco_document_ranking\", \"benchmark_type\": \"base_rag\", \"sub_benchmark\": \"document_ranking\"}" } ``` **Dev** (`expected_output` is a JSON-encoded list of relevant document ids): ```json { "input": "does xpress bet charge to deposit money in your account", "expected_output": "[\"D1987644\"]", "metadata": "{\"query_id\": \"174249\", \"split\": \"dev\", \"benchmark_name\": \"msmarco_document_ranking\", \"benchmark_type\": \"base_rag\", \"sub_benchmark\": \"document_ranking\"}" } ``` **Test** (labels may be withheld; example shows an empty list): ```json { "input": "how to display how.close you are to.cell.tower", "expected_output": "[]", "metadata": "{\"query_id\": \"355339\", \"split\": \"test\", \"benchmark_name\": \"msmarco_document_ranking\", \"benchmark_type\": \"base_rag\", \"sub_benchmark\": \"document_ranking\"}" } ``` ## References ### Foundational MS MARCO paper (cite when using MS MARCO-derived data) **Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, Tong Wang.** *MS MARCO: A Human Generated MAchine Reading COmprehension Dataset.* arXiv:1611.09268, 2016. **Abstract (short):** MS MARCO introduces a large-scale machine reading comprehension dataset built from real Bing queries, with human-generated answers and millions of passages from retrieved web documents. The work defines multiple tasks of varying difficulty, including passage ranking—establishing MS MARCO as a benchmark for realistic, large-scale QA and IR research. - Paper: [arXiv:1611.09268](https://arxiv.org/abs/1611.09268) ```bibtex @article{bajaj2016ms, title={Ms marco: A human generated machine reading comprehension dataset}, author={Bajaj, Payal and Campos, Daniel and Craswell, Nick and Deng, Li and Gao, Jianfeng and Liu, Xiaodong and Majumder, Rangan and McNamara, Andrew and Mitra, Bhaskar and Nguyen, Tri and others}, journal={arXiv preprint arXiv:1611.09268}, year={2016} } ``` ### Official dataset and leaderboard documentation - [MS MARCO — Datasets for Document and Passage Ranking Leaderboards](https://microsoft.github.io/msmarco/Datasets.html) — corpus files, qrels, submission formats, and task description. - [MS MARCO — Submission / evaluation](https://microsoft.github.io/msmarco/Submission.html) — ranking submission conventions. - [TREC Deep Learning Track](https://microsoft.github.io/msmarco/TREC-Deep-Learning) — blind evaluation and community benchmarks related to MS MARCO ranking tasks. ### Related code and corpora (Microsoft) - [microsoft/MSMARCO-Document-Ranking](https://github.com/microsoft/MSMARCO-Document-Ranking) — pointers and tooling around the document ranking collection. ## Terms and licensing The **underlying MS MARCO data** is subject to Microsoft’s **terms for non-commercial research** as published on the official MS MARCO site; review the **Terms and Conditions** on [the official datasets page](https://microsoft.github.io/msmarco/Datasets.html) before use in products or redistributions. This Hugging Face dataset card describes the **CTERA-formatted** release; when publishing results, cite **MS MARCO** appropriately and follow the original **license / usage** constraints for the source corpus and judgments. ## Acknowledgments Dataset packaging for this repository is maintained by **CTERA**. **MS MARCO** is provided by **Microsoft** and the broader IR community; see the official site for credits and contact information.--- 许可证：其他 任务类别： - 文本检索 语言： - 英语 标签： - 信息检索 - 文档排序 - msmarco - 基准测试 - 检索 美观名称：MS MARCO 文档排序（CTERA格式） 规模类别： - "100K<n<1M" --- # MS MARCO 文档排序 ## 数据集描述与背景 **MS MARCO（微软机器阅读理解，MicroSoft MAchine Reading COmprehension）** 是一个大规模数据集，最初为机器阅读理解与问答任务提出。随着时间推移，它已成为拥有充足训练数据的**信息检索**标准基准：包含数十万条带有人类相关性标注的查询，且对齐真实网络搜索行为。 该**文档排序**赛道使用源自与MS MARCO段落排序相同生态的**文档级语料库**。官方资源描述的语料库规模约为**~320万篇文档**，训练查询量达数十万级，同时提供**开发集**与**排行榜测试集**查询集，以及用于训练与开发的**TREC风格qrels（查询-相关性标注）**。在训练阶段，**段落级标注会被映射至文档ID**，其假设为：包含被标注为相关段落的文档即视为相关文档，这一设计支持面向段落与面向文档的检索研究间的迁移。 本Hugging Face数据集是该任务的**CTERA封装视图**：每一行均将**自然语言查询**与适用于检索与**检索增强生成（Retrieval-Augmented Generation，RAG）**基准测试的**结构化标注/元数据**配对（详见下文[数据字段](#data-fields)）。 官方MS MARCO排序资源（语料库、qrels、排行榜）：[MS MARCO — Datasets for Document and Passage Ranking](https://microsoft.github.io/msmarco/Datasets.html) ## 任务：文档排序/检索 本数据集支持**即席文档检索/排序**：给定一个**查询**，系统需根据**相关性**对集合中的文档进行排序。在研究场景中，该任务通常分为两类： - **全量排序（检索）**：从**完整文档集合**中打分或检索（官方提交要求每个查询最多处理有限数量的文档，例如MS MARCO文档排序设置中最多为**100篇**）。 - **Top-k重排序**：对固定候选列表进行重排序（例如第一阶段检索器得到的**Top-100候选**）——这是“先检索、后重排序”的常见生产模式。 MS MARCO排序排行榜的**评估指标**通常以文档排序任务的**MRR@10（排名10处的平均倒数排名）**为主，同时可辅以标准TREC风格分析（如需了解特定赛事使用的精确指标定义，请参阅官方排行榜与TREC深度学习赛道资料）。 ## 数据字段 Parquet分割文件包含三列： | 列名 | 描述 | |--------|-------------| | `input` | 查询文本（字符串类型）。 | | `expected_output` | 包含相关性信息的JSON字符串；**格式因分割集而异**（详见示例）。 | | `metadata` | 包含标识符与基准标签的JSON字符串（`benchmark_name`、`split`、`query_id`等）。 | **分割集**：提供`train`（训练集）、`dev`（开发集）与`test`（测试集）。**测试集**的`expected_output`中可能使用**预留标签**或**空值**以支持盲评估流程——请参阅下方示例。 ## 示例 **训练集**（`expected_output`以JSON形式包含查询ID与被标注为相关的文档ID）： json { "input": ")what was the immediate impact of the success of the manhattan project?", "expected_output": "{"qid": "1185869", "qrels": ["D59219"]}", "metadata": "{"query_id": "1185869", "split": "train", "benchmark_name": "msmarco_document_ranking", "benchmark_type": "base_rag", "sub_benchmark": "document_ranking"}" } **开发集**（`expected_output`为JSON编码的相关文档ID列表）： json { "input": "does xpress bet charge to deposit money in your account", "expected_output": "["D1987644"]", "metadata": "{"query_id": "174249", "split": "dev", "benchmark_name": "msmarco_document_ranking", "benchmark_type": "base_rag", "sub_benchmark": "document_ranking"}" } **测试集**（标签可能被隐藏；示例展示为空列表）： json { "input": "how to display how.close you are to.cell.tower", "expected_output": "[]", "metadata": "{"query_id": "355339", "split": "test", "benchmark_name": "msmarco_document_ranking", "benchmark_type": "base_rag", "sub_benchmark": "document_ranking"}" } ## 参考文献 ### 基础MS MARCO论文（使用MS MARCO衍生数据时需引用） **Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, Tong Wang.** *MS MARCO: A Human Generated MAchine Reading COmprehension Dataset.* arXiv:1611.09268, 2016. **简短摘要**：MS MARCO推出了一个大规模机器阅读理解数据集，其数据源自真实的必应查询，包含人类生成的答案与来自检索到的网络文档的数百万篇段落。该工作定义了多个难度各异的任务，包括段落排序，使MS MARCO成为面向真实场景的大规模问答与信息检索研究的基准。 - 论文链接：[arXiv:1611.09268](https://arxiv.org/abs/1611.09268) bibtex @article{bajaj2016ms, title={Ms marco: A human generated machine reading comprehension dataset}, author={Bajaj, Payal and Campos, Daniel and Craswell, Nick and Deng, Li and Gao, Jianfeng and Liu, Xiaodong and Majumder, Rangan and McNamara, Andrew and Mitra, Bhaskar and Nguyen, Tri and others}, journal={arXiv preprint arXiv:1611.09268}, year={2016} } ### 官方数据集与排行榜文档 - [MS MARCO — Datasets for Document and Passage Ranking Leaderboards](https://microsoft.github.io/msmarco/Datasets.html) — 语料库文件、qrels、提交格式与任务说明。 - [MS MARCO — Submission / evaluation](https://microsoft.github.io/msmarco/Submission.html) — 排序提交规范。 - [TREC Deep Learning Track](https://microsoft.github.io/msmarco/TREC-Deep-Learning) — 与MS MARCO排序任务相关的盲评估与社区基准测试。 ### 相关代码与语料库（微软官方） - [microsoft/MSMARCO-Document-Ranking](https://github.com/microsoft/MSMARCO-Document-Ranking) — 文档排序语料库的相关指针与工具。 ## 条款与许可 **底层MS MARCO数据**需遵循微软官方MS MARCO网站发布的**非商业研究条款**；在用于产品或再分发前，请查阅[官方数据集页面](https://microsoft.github.io/msmarco/Datasets.html)上的**条款与条件**。 本Hugging Face数据集卡片描述的是**CTERA格式**的发布版本；在发布研究结果时，请合理引用**MS MARCO**，并遵循源语料库与标注的原始**许可/使用约束**。 ## 致谢 本仓库的数据集封装由**CTERA**维护。**MS MARCO**由**微软**与更广泛的信息检索社区提供；如需致谢与联系方式，请参阅官方网站。