facebook/wiki_dpr
收藏资源简介：
这是用于评估Dense Passage Retrieval (DPR)模型的维基百科分割数据集。它包含2100万条来自维基百科的段落及其DPR嵌入。维基百科文章被分割为多个不重叠的100字文本块作为段落。数据集基于2018年12月20日的维基百科转储。有两种类型的DPR嵌入，分别基于不同的模型：`nq`（在Natural Questions数据集上训练）和`multiset`（在多个数据集上训练）。此外，可以从嵌入中创建FAISS索引：`exact`（精确索引，高内存使用）、`compressed`（压缩索引，近似，低内存使用）和`no_index`（无索引）。还可以生成不带嵌入的数据集：`no_embeddings`。
This is a segmented Wikipedia dataset intended for evaluating Dense Passage Retrieval (DPR) models. It comprises 21 million Wikipedia paragraphs paired with their corresponding DPR embeddings. Wikipedia articles are split into non-overlapping 100-word text chunks that serve as the paragraphs in this dataset. This dataset is built upon the Wikipedia dump dated December 20, 2018. There are two types of DPR embeddings based on different training models: `nq` (trained on the Natural Questions dataset) and `multiset` (trained on multiple datasets). Additionally, three types of FAISS indexes can be created from these embeddings: `exact` (exact index with high memory usage), `compressed` (compressed approximate index with low memory usage), and `no_index` (no index). Furthermore, a dataset without embeddings, labeled `no_embeddings`, can be generated.
数据集概述
基本信息
- 数据集名称: Wiki-DPR
- 语言: 英语
- 许可证: CC-BY-NC-4.0
- 多语言性: 多语言
- 数据集大小: 10M<n<100M
- 源数据: 原始数据
- 任务类别:
- 填空
- 文本生成
- 任务ID:
- 语言建模
- 掩码语言建模
- 标签: 文本搜索
数据集结构
配置详情
-
配置名称: psgs_w100.nq.exact
- 特征:
id: 字符串text: 字符串title: 字符串embeddings: 浮点数序列
- 分割:
train:- 字节数: 78419248156
- 样本数: 21015300
- 下载大小: 85288677114
- 数据集大小: 78419248156
- 特征:
-
配置名称: psgs_w100.nq.compressed
- 特征:
id: 字符串text: 字符串title: 字符串embeddings: 浮点数序列
- 分割:
train:- 字节数: 78419248156
- 样本数: 21015300
- 下载大小: 85288677114
- 数据集大小: 78419248156
- 特征:
-
配置名称: psgs_w100.nq.no_index
- 特征:
id: 字符串text: 字符串title: 字符串embeddings: 浮点数序列
- 分割:
train:- 字节数: 78419248156
- 样本数: 21015300
- 下载大小: 85288677114
- 数据集大小: 78419248156
- 特征:
-
配置名称: psgs_w100.multiset.exact
- 特征:
id: 字符串text: 字符串title: 字符串embeddings: 浮点数序列
- 分割:
train:- 字节数: 78419248156
- 样本数: 21015300
- 下载大小: 85289275855
- 数据集大小: 78419248156
- 特征:
-
配置名称: psgs_w100.multiset.compressed
- 特征:
id: 字符串text: 字符串title: 字符串embeddings: 浮点数序列
- 分割:
train:- 字节数: 78419248156
- 样本数: 21015300
- 下载大小: 85289275855
- 数据集大小: 78419248156
- 特征:
-
配置名称: psgs_w100.multiset.no_index
- 特征:
id: 字符串text: 字符串title: 字符串embeddings: 浮点数序列
- 分割:
train:- 字节数: 78419248156
- 样本数: 21015300
- 下载大小: 85289275855
- 数据集大小: 78419248156
- 特征:
数据实例
每个实例包含最多100个单词的段落，以及该段落来自的维基百科页面的标题和DPR嵌入（一个768维的向量）。
示例
json { "id": "1", "text": "Aaron Aaron ( or ; "Ahärôn") is a prophet, high priest, and the brother of Moses in the Abrahamic religions. Knowledge of Aaron, along with his brother Moses, comes exclusively from religious texts, such as the Bible and Quran. The Hebrew Bible relates that, unlike Moses, who grew up in the Egyptian royal court, Aaron and his elder sister Miriam remained with their kinsmen in the eastern border-land of Egypt (Goshen). When Moses first confronted the Egyptian king about the Israelites, Aaron served as his brothers spokesman ("prophet") to the Pharaoh. Part of the Law (Torah) that Moses received from", "title": "Aaron", "embeddings": [-0.07233893871307373, 0.48035329580307007, 0.18650995194911957, -0.5287084579467773, -0.37329429388046265, 0.37622880935668945, 0.25524479150772095, ... -0.336689829826355, 0.6313082575798035, -0.7025573253631592] }
数据字段
所有分割的数据字段相同：
id: 字符串特征text: 字符串特征title: 字符串特征embeddings: 浮点数序列特征
数据分割
| 名称 | 训练样本数 |
|---|---|
| psgs_w100.multiset.compressed | 21015300 |
| psgs_w100.multiset.exact | 21015300 |
| psgs_w100.multiset.no_index | 21015300 |
| psgs_w100.nq.compressed | 21015300 |
| psgs_w100.nq.exact | 21015300 |
