kilt_wikipedia

# Dataset Card for "kilt_wikipedia"

## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Homepage:** [https://github.com/facebookresearch/KILT](https://github.com/facebookresearch/KILT)
- **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Paper:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Size of downloaded dataset files:** 37.32 GB
- **Size of the generated dataset:** 29.37 GB
- **Total amount of disk used:** 66.69 GB

### Dataset Summary

KILT-Wikipedia: Wikipedia pre-processed for KILT.

### Supported Tasks and Leaderboards

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Languages

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Dataset Structure

### Data Instances

#### 2019-08-01

- **Size of downloaded dataset files:** 37.32 GB
- **Size of the generated dataset:** 29.37 GB
- **Total amount of disk used:** 66.69 GB

An example of 'full' looks as follows.
```
{
"anchors": {
"end": [],
"href": [],
"paragraph_id": [],
"start": [],
"text": [],
"wikipedia_id": [],
"wikipedia_title": []
},
"categories": "",
"history": {
"pageid": 0,
"parentid": 0,
"pre_dump": true,
"revid": 0,
"timestamp": "",
"url": ""
},
"kilt_id": "",
"text": {
"paragraph": []
},
"wikidata_info": {
"aliases": {
"alias": []
},
"description": "",
"enwikiquote_title": "",
"wikidata_id": "",
"wikidata_label": "",
"wikipedia_title": ""
},
"wikipedia_id": "",
"wikipedia_title": ""
}
```

### Data Fields

The data fields are the same among all splits.

#### 2019-08-01
- `kilt_id`: a `string` feature.
- `wikipedia_id`: a `string` feature.
- `wikipedia_title`: a `string` feature.
- `text`: a dictionary feature containing:
- `paragraph`: a `string` feature.
- `anchors`: a dictionary feature containing:
- `paragraph_id`: a `int32` feature.
- `start`: a `int32` feature.
- `end`: a `int32` feature.
- `text`: a `string` feature.
- `href`: a `string` feature.
- `wikipedia_title`: a `string` feature.
- `wikipedia_id`: a `string` feature.
- `categories`: a `string` feature.
- `description`: a `string` feature.
- `enwikiquote_title`: a `string` feature.
- `wikidata_id`: a `string` feature.
- `wikidata_label`: a `string` feature.
- `wikipedia_title`: a `string` feature.
- `aliases`: a dictionary feature containing:
- `alias`: a `string` feature.
- `pageid`: a `int32` feature.
- `parentid`: a `int32` feature.
- `revid`: a `int32` feature.
- `pre_dump`: a `bool` feature.
- `timestamp`: a `string` feature.
- `url`: a `string` feature.

### Data Splits

| name | full |
|----------|------:|
|2019-08-01|5903530|

## Dataset Creation

### Curation Rationale

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

#### Who are the source language producers?

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Annotations

#### Annotation process

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

#### Who are the annotators?

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Personal and Sensitive Information

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Discussion of Biases

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Other Known Limitations

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Additional Information

### Dataset Curators

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Licensing Information

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Citation Information

```
@inproceedings{fb_kilt,
author = {Fabio Petroni and
Aleksandra Piktus and
Angela Fan and
Patrick Lewis and
Majid Yazdani and
Nicola De Cao and
James Thorne and
Yacine Jernite and
Vassilis Plachouras and
Tim Rockt"aschel and
Sebastian Riedel},
title = {{KILT:} a {B}enchmark for {K}nowledge {I}ntensive {L}anguage {T}asks},
journal = {CoRR},
archivePrefix = {arXiv},
year = {2020},

```

### Contributions

Thanks to [@thomwolf](https://github.com/thomwolf), [@yjernite](https://github.com/yjernite) for adding this dataset.

# 数据集卡片："kilt_wikipedia"

## 目录
- [数据集描述](#dataset-description)
- [数据集概览](#dataset-summary)
- [支持任务与排行榜](#supported-tasks-and-leaderboards)
- [语言](#languages)
- [数据集结构](#dataset-structure)
- [数据实例](#data-instances)
- [数据字段](#data-fields)
- [数据划分](#data-splits)
- [数据集构建](#dataset-creation)
- [遴选依据](#curation-rationale)
- [源数据](#source-data)
- [标注信息](#annotations)
- [个人与敏感信息](#personal-and-sensitive-information)
- [数据集使用注意事项](#considerations-for-using-the-data)
- [数据集的社会影响](#social-impact-of-dataset)
- [偏差讨论](#discussion-of-biases)
- [其他已知局限性](#other-known-limitations)
- [附加信息](#additional-information)
- [数据集遴选者](#dataset-curators)
- [授权信息](#licensing-information)
- [引用信息](#citation-information)
- [贡献者](#contributions)

## 数据集描述
- **主页:** [https://github.com/facebookresearch/KILT](https://github.com/facebookresearch/KILT)
- **代码仓库:** [更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **相关论文:** [更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **联系人:** [更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **下载数据集文件大小:** 37.32 GB
- **生成后数据集大小:** 29.37 GB
- **总磁盘占用:** 66.69 GB

### 数据集概览
KILT-维基百科：针对KILT预处理后的维基百科数据集。

### 支持任务与排行榜
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### 语言
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## 数据集结构
### 数据实例
#### 2019-08-01
- **下载数据集文件大小:** 37.32 GB
- **生成后数据集大小:** 29.37 GB
- **总磁盘占用:** 66.69 GB

"full"类型的示例如下：

{
"anchors": {
"end": [],
"href": [],
"paragraph_id": [],
"start": [],
"text": [],
"wikipedia_id": [],
"wikipedia_title": []
},
"categories": "",
"history": {
"pageid": 0,
"parentid": 0,
"pre_dump": true,
"revid": 0,
"timestamp": "",
"url": ""
},
"kilt_id": "",
"text": {
"paragraph": []
},
"wikidata_info": {
"aliases": {
"alias": []
},
"description": "",
"enwikiquote_title": "",
"wikidata_id": "",
"wikidata_label": "",
"wikipedia_title": ""
},
"wikipedia_id": "",
"wikipedia_title": ""
}

### 数据字段
所有划分下的数据字段均保持一致。
#### 2019-08-01
- `kilt_id`: 字符串类型特征
- `wikipedia_id`: 字符串类型特征
- `wikipedia_title`: 字符串类型特征
- `text`: 字典类型特征，包含以下子字段：
- `paragraph`: 字符串类型特征
- `anchors`: 字典类型特征，包含以下子字段：
- `paragraph_id`: int32类型特征
- `start`: int32类型特征
- `end`: int32类型特征
- `text`: 字符串类型特征
- `href`: 字符串类型特征
- `wikipedia_title`: 字符串类型特征
- `wikipedia_id`: 字符串类型特征
- `categories`: 字符串类型特征
- `description`: 字符串类型特征
- `enwikiquote_title`: 字符串类型特征
- `wikidata_id`: 字符串类型特征
- `wikidata_label`: 字符串类型特征
- `wikipedia_title`: 字符串类型特征
- `aliases`: 字典类型特征，包含以下子字段：
- `alias`: 字符串类型特征
- `pageid`: int32类型特征
- `parentid`: int32类型特征
- `revid`: int32类型特征
- `pre_dump`: 布尔类型特征
- `timestamp`: 字符串类型特征
- `url`: 字符串类型特征

### 数据划分
| 名称 | full |
|----------|------:|
|2019-08-01|5903530|

## 数据集构建
### 遴选依据
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### 源数据
#### 初始数据收集与规范化
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

#### 源语言生产者是谁？
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### 标注信息
#### 标注流程
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

#### 标注者是谁？
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### 个人与敏感信息
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## 数据集使用注意事项
### 数据集的社会影响
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### 偏差讨论
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### 其他已知局限性
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## 附加信息
### 数据集遴选者
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### 授权信息
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### 引用信息

@inproceedings{fb_kilt,
author = {Fabio Petroni and
Aleksandra Piktus and
Angela Fan and
Patrick Lewis and
Majid Yazdani and
Nicola De Cao and
James Thorne and
Yacine Jernite and
Vassilis Plachouras and
Tim Rockt"aschel and
Sebastian Riedel},
title = {{KILT:} a {B}enchmark for {K}nowledge {I}ntensive {L}anguage {T}asks},
journal = {CoRR},
archivePrefix = {arXiv},
year = {2020},

### 贡献者
感谢 [@thomwolf](https://github.com/thomwolf)、[@yjernite](https://github.com/yjernite) 添加此数据集。