irds/natural-questions

`natural-questions`数据集由`ir-datasets`包提供，主要用于文本检索任务。该数据集包含28,390,850个文档，每个文档包含多个字段，如`doc_id`、`text`、`html`等。用户可以通过`datasets`库加载数据集，并提供了示例代码。该数据集的相关研究论文为《Natural Questions: a Benchmark for Question Answering Research》。

The Natural Questions dataset is provided through the `ir-datasets` package, and is mainly designed for text retrieval tasks. It consists of 28,390,850 documents, each with multiple fields including `doc_id`, `text`, `html`, and others. The dataset can be loaded via the `datasets` library, with corresponding example code available. The associated research paper for this dataset is *Natural Questions: a Benchmark for Question Answering Research*.

原始信息汇总

数据集概述

数据集名称

natural-questions

数据集来源

由 ir-datasets 包提供。

数据集内容

• 数据类型: docs (文档，即语料库)
• 文档数量: 28,390,850

数据集结构

每个文档记录包含以下字段：

• doc_id
• text
• html
• start_byte
• end_byte
• start_token
• end_token
• document_title
• document_url
• parent_doc_id

使用方法

通过以下代码加载数据集： python from datasets import load_dataset

docs = load_dataset(irds/natural-questions, docs) for record in docs: record # 包含上述字段

引用信息

@article{Kwiatkowski2019Nq, title = {Natural Questions: a Benchmark for Question Answering Research}, author = {Tom Kwiatkowski and Jennimaria Palomaki and Olivia Redfield and Michael Collins and Ankur Parikh and Chris Alberti and Danielle Epstein and Illia Polosukhin and Matthew Kelcey and Jacob Devlin and Kenton Lee and Kristina N. Toutanova and Llion Jones and Ming-Wei Chang and Andrew Dai and Jakob Uszkoreit and Quoc Le and Slav Petrov}, year = {2019}, journal = {TACL} }

搜集汇总

背景与挑战

背景概述

该数据集是 'ir-datasets' 包中的 'natural-questions'，主要用于文本检索任务。它包含一个庞大的文档集合，数量达到28,390,850个，旨在为问答研究提供一个基准数据集。