wikipedia

# Dataset Card for Wikimedia Wikipedia

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Homepage:** [https://dumps.wikimedia.org](https://dumps.wikimedia.org)
- **Repository:**
- **Paper:**
- **Point of Contact:**

### Dataset Summary

Wikipedia dataset containing cleaned articles of all languages.

The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/)
with one subset per language, each containing a single train split.

Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).

All language subsets have already been processed for recent dump, and you can load them per date and language this way:
```python
from datasets import load_dataset

ds = load_dataset("wikimedia/wikipedia", "20231101.en")
```

#### Data Visualization
Click the [Nomic Atlas](https://atlas.nomic.ai/map/475c26d7-b142-4795-9887-02b6eeb18dc0/0d312be6-a3bb-4586-b6b7-53dcd0cbefa5) map below to visualize the 6.4 million samples in the `20231101.en` split.

<a href="https://atlas.nomic.ai/map/475c26d7-b142-4795-9887-02b6eeb18dc0/0d312be6-a3bb-4586-b6b7-53dcd0cbefa5">
<img src="https://cdn-uploads.huggingface.co/production/uploads/6480c476cacb1c4a0696eeb8/sZNN6Vubc0Oue83vKaJUu.webp" alt="Nomic-Atlas Wikipedia Map" width="25%"/>
</a>

### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling.

### Languages

You can find the list of languages here: https://meta.wikimedia.org/wiki/List_of_Wikipedias

## Dataset Structure

### Data Instances

An example looks as follows:
```
{'id': '1',
'url': 'https://simple.wikipedia.org/wiki/April',
'title': 'April',
'text': 'April is the fourth month...'
}
```

### Data Fields

The data fields are the same among all configurations:
- `id` (`str`): ID of the article.
- `url` (`str`): URL of the article.
- `title` (`str`): Title of the article.
- `text` (`str`): Text content of the article.

### Data Splits

All configurations contain a single `train` split.

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

The dataset is built from the Wikipedia dumps: https://dumps.wikimedia.org

You can find the full list of languages and dates here: https://dumps.wikimedia.org/backup-index.html

The articles have been parsed using the [`mwparserfromhell`](https://mwparserfromhell.readthedocs.io) tool.

When uploading the data files for the 20231101 dump, we noticed that the Wikimedia Dumps website does not contain this date dump
for the "bbc", "dga", nor "zgh" Wikipedias. We have reported the issue to the Wikimedia Phabricator: https://phabricator.wikimedia.org/T351761

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

Copyright licensing information: https://dumps.wikimedia.org/legal.html

All original textual content is licensed under the [GNU Free Documentation License](https://www.gnu.org/licenses/fdl-1.3.html) (GFDL)
and the [Creative Commons Attribution-Share-Alike 3.0 License](https://creativecommons.org/licenses/by-sa/3.0/).
Some text may be available only under the Creative Commons license; see their [Terms of Use](https://foundation.wikimedia.org/wiki/Policy:Terms_of_Use) for details.
Text written by some authors may be released under additional licenses or into the public domain.

### Citation Information

```
@ONLINE{wikidump,
author = "Wikimedia Foundation",
title = "Wikimedia Downloads",
url = "https://dumps.wikimedia.org"
}
```

# Wikimedia Wikipedia 数据集卡片

## 目录
- [目录](#table-of-contents)
- [数据集描述](#dataset-description)
- [数据集概述](#dataset-summary)
- [支持任务与基准榜单](#supported-tasks-and-leaderboards)
- [支持语言](#languages)
- [数据集结构](#dataset-structure)
- [数据实例](#data-instances)
- [数据字段](#data-fields)
- [数据划分](#data-splits)
- [数据集构建](#dataset-creation)
- [构建初衷](#curation-rationale)
- [源数据](#source-data)
- [标注信息](#annotations)
- [个人与敏感信息](#personal-and-sensitive-information)
- [数据集使用注意事项](#considerations-for-using-the-data)
- [数据集的社会影响](#social-impact-of-dataset)
- [偏差讨论](#discussion-of-biases)
- [其他已知局限](#other-known-limitations)
- [附加信息](#additional-information)
- [数据集维护者](#dataset-curators)
- [许可信息](#licensing-information)
- [引用信息](#citation-information)
- [贡献](#contributions)

## 数据集描述

- **主页：** [https://dumps.wikimedia.org](https://dumps.wikimedia.org)
- **代码仓库：**
- **相关论文：**
- **联系方式：**

### 数据集概述
本数据集包含所有语言的清洗后Wikipedia文章。
本数据集基于Wikipedia转储文件（https://dumps.wikimedia.org/）构建，按语言划分子集，每个子集仅包含一个训练划分。
每条数据对应一篇完整的Wikipedia文章，已通过清洗去除Markdown格式与不必要的章节（如参考文献章节等）。

所有语言子集均已针对近期转储文件完成处理，你可以通过如下方式按日期与语言加载数据集：
python
from datasets import load_dataset

ds = load_dataset("wikimedia/wikipedia", "20231101.en")

#### 数据可视化
点击下方的[Nomic Atlas](https://atlas.nomic.ai/map/475c26d7-b142-4795-9887-02b6eeb18dc0/0d312be6-a3bb-4586-b6b7-53dcd0cbefa5)地图，即可可视化`20231101.en`划分中的640万条样本。

<a href="https://atlas.nomic.ai/map/475c26d7-b142-4795-9887-02b6eeb18dc0/0d312be6-a3bb-4586-b6b7-53dcd0cbefa5">
<img src="https://cdn-uploads.huggingface.co/production/uploads/6480c476cacb1c4a0696eeb8/sZNN6Vubc0Oue83vKaJUu.webp" alt="Nomic-Atlas Wikipedia 地图" width="25%"/>
</a>

### 支持任务与基准榜单
本数据集通常用于语言建模任务。

### 支持语言
你可在此处查询完整语言列表：https://meta.wikimedia.org/wiki/List_of_Wikipedias

## 数据集结构

### 数据实例
数据实例示例如下：

{'id': '1',
'url': 'https://simple.wikipedia.org/wiki/April',
'title': 'April',
'text': 'April is the fourth month...'
}

### 数据字段
所有数据集配置的字段均保持一致：
- `id`（字符串类型）：文章的唯一标识符。
- `url`（字符串类型）：文章的原始链接。
- `title`（字符串类型）：文章的标题。
- `text`（字符串类型）：文章的文本内容。

### 数据划分
所有数据集配置均仅包含一个`train`（训练）划分。

## 数据集构建

### 构建初衷
[需要更多信息]

### 源数据

#### 初始数据收集与标准化
本数据集基于Wikipedia转储文件（https://dumps.wikimedia.org/）构建。
你可在此处查询完整的语言与转储日期列表：https://dumps.wikimedia.org/backup-index.html
文章内容已通过[`mwparserfromhell`](https://mwparserfromhell.readthedocs.io)工具进行解析。
在上传20231101版转储文件的数据时，我们发现Wikimedia转储网站未提供“bbc”“dga”以及“zgh”三种语言维基百科的该日期转储文件。我们已就此问题向Wikimedia Phabricator提交反馈：https://phabricator.wikimedia.org/T351761

#### 源语言内容的创作者是谁？
[需要更多信息]

### 标注信息

#### 标注流程
[需要更多信息]

#### 标注者是谁？
[需要更多信息]

### 个人与敏感信息
[需要更多信息]

## 数据集使用注意事项

### 数据集的社会影响
[需要更多信息]

### 偏差讨论
[需要更多信息]

### 其他已知局限
[需要更多信息]

## 附加信息

### 数据集维护者
[需要更多信息]

### 许可信息
版权许可信息：https://dumps.wikimedia.org/legal.html
所有原始文本内容均遵循[GNU Free Documentation License (GFDL)](https://www.gnu.org/licenses/fdl-1.3.html)与[Creative Commons Attribution-Share-Alike 3.0 License (CC BY-SA 3.0)](https://creativecommons.org/licenses/by-sa/3.0/)许可。部分文本可能仅可在知识共享许可协议下使用，详细信息请参阅其[使用条款](https://foundation.wikimedia.org/wiki/Policy:Terms_of_Use)。部分由特定作者撰写的文本可能遵循其他许可协议或已进入公有领域。

### 引用信息

@ONLINE{wikidump,
author = "Wikimedia Foundation",
title = "Wikimedia Downloads",
url = "https://dumps.wikimedia.org"
}

### 贡献
[需要更多信息]