ABC321000/open-web-math

---
dataset_info:
features:
- name: url
dtype: string
- name: text
dtype: string
- name: date
dtype: string
- name: metadata
dtype: string
splits:
- name: train
num_bytes: 56651995057
num_examples: 6315233
download_size: 16370689925
dataset_size: 56651995057
license: odc-by
task_categories:
- text-generation
language:
- en
pretty_name: OpenWebMath
size_categories:
- 10B<n<100B
---

<img src="imgs/OpenWebMath-left.png" width="300">

[Keiran Paster](https://keirp.com)\*, [Marco Dos Santos](https://marco-dossantos.github.io/)\*, [Zhangir Azerbayev](https://zhangir-azerbayev.github.io/), [Jimmy Ba](https://jimmylba.github.io/)

[GitHub ](https://github.com/keirp/OpenWebMath) | [ArXiv](https://arxiv.org/abs/2310.06786)
| [PDF](https://arxiv.org/pdf/2310.06786.pdf)

**OpenWebMath** is a dataset containing the majority of the high-quality, mathematical text from the internet. It is filtered and extracted from over 200B HTML files on Common Crawl down to a set of **6.3 million documents** containing a total of **14.7B tokens**. OpenWebMath is intended for use in _pretraining_ and _finetuning_ large language models.

You can download the dataset using Hugging Face:

```python
from datasets import load_dataset
ds = load_dataset("open-web-math/open-web-math")
```

# OpenWebMath Contents

The dataset is structured as follows:

```python
{
"text": ..., # document text.
"url": ..., # document url.
"date": ..., # date the page was crawled.
"metadata": ..., # JSON containing information from the extraction process.
}
```

OpenWebMath contains documents from over 130k different domains, including data from forums, educational pages, and blogs. The dataset contains documents covering mathematics, physics, statistics, computer science, and more. The following table shows the most common domains in OpenWebMath by character count.

| Domain | # Characters | % Characters |
| ----------------- | ------------- | ------------ |
| stackexchange.com | 4,655,132,784 | 9.55% |
| nature.com | 1,529,935,838 | 3.14% |
| wordpress.com | 1,294,166,938 | 2.66% |
| physicsforums.com | 1,160,137,919 | 2.38% |
| github.io | 725,689,722 | 1.49% |
| zbmath.org | 620,019,503 | 1.27% |
| wikipedia.org | 618,024,754 | 1.27% |
| groundai.com | 545,214,990 | 1.12% |
| blogspot.com | 520,392,333 | 1.07% |
| mathoverflow.net | 499,102,560 | 1.02% |

# OpenWebMath Pipeline

<img src="imgs/pipeline.png" alt="Overview of the OpenWebMath Pipeline">

OpenWebMath builds on the massive [Common Crawl](https://commoncrawl.org/) dataset, which contains over 200B HTML documents. We filtered the data to only include documents that are: (1) in English, (2) contain mathematical content, and (3) are of high quality. We also put a strong emphasis on extracting LaTeX content from the HTML documents as well as reducing boilerplate in comparison to other web datasets.

The OpenWebMath pipeline consists of five steps:

1. **Prefiltering HTML Documents**:
- We apply a simple prefilter to all HTML documents in Common Crawl in order to skip documents without mathematical content to unnecessary processing time.
2. **Text Extraction**:
- Extract text, including LaTeX content, from the HTML documents while removing boilerplate.
3. **Content Classification and Filtering**:
- Apply a [FastText language identification model](https://fasttext.cc/docs/en/language-identification.html) to keep only English documents.
- Filter high perplexity documents using a [KenLM](https://github.com/kpu/kenlm) model trained on [Proof-Pile](https://huggingface.co/datasets/hoskinson-center/proof-pile).
- Filter non-mathematical documents using our own _MathScore_ model.
4. **Deduplication**:
- Deduplicate the dataset using SimHash in [text-dedup](https://github.com/ChenghaoMou/text-dedup).
5. **Manual Inspection**:
- Inspect the documents gathered from previous steps and remove low quality pages.

For a detailed discussion on the processing pipeline, please refer to our paper.

# License

OpenWebMath is made available under an ODC-By 1.0 license; users should also abide by the CommonCrawl ToU: [https://commoncrawl.org/terms-of-use/](https://commoncrawl.org/terms-of-use/). We do not alter the license of any of the underlying data.

# Citation Information

```
@misc{paster2023openwebmath,
title={OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text},
author={Keiran Paster and Marco Dos Santos and Zhangir Azerbayev and Jimmy Ba},
year={2023},
eprint={2310.06786},
archivePrefix={arXiv},
primaryClass={cs.AI}
}
```

dataset_info:
特征：
- 名称：url，数据类型：字符串
- 名称：text，数据类型：字符串
- 名称：date，数据类型：字符串
- 名称：metadata，数据类型：字符串
划分：
- 名称：train，字节数：56651995057，样本数：6315233
下载大小：16370689925
数据集总大小：56651995057
许可协议：odc-by
任务类别：
- 文本生成
语言：
- en
美观名称：OpenWebMath
规模类别：
- 10B<n<100B

<img src="imgs/OpenWebMath-left.png" width="300">

[Keiran Paster](https://keirp.com)*, [Marco Dos Santos](https://marco-dossantos.github.io/)*, [Zhangir Azerbayev](https://zhangir-azerbayev.github.io/), [Jimmy Ba](https://jimmylba.github.io/)

[GitHub](https://github.com/keirp/OpenWebMath) | [ArXiv](https://arxiv.org/abs/2310.06786) | [PDF](https://arxiv.org/pdf/2310.06786.pdf)

**OpenWebMath** 是一个收录互联网上绝大多数高质量数学文本的数据集。其从通用爬虫（Common Crawl）的2000亿余份HTML文件中筛选提取而来，最终包含631.5万份文档，总计147亿Token（Token）。OpenWebMath旨在用于大语言模型（Large Language Model, LLM）的预训练与微调。

你可以通过Hugging Face下载该数据集：
python
from datasets import load_dataset
ds = load_dataset("open-web-math/open-web-math")

# OpenWebMath 数据内容
该数据集的结构如下：
python
{
"text": ..., # 文档文本
"url": ..., # 文档的统一资源定位符
"date": ..., # 页面被爬取的日期
"metadata": ..., # 包含提取过程相关信息的JSON数据
}

OpenWebMath包含来自13万余个不同域名的文档，涵盖论坛、教育页面与博客等来源，内容覆盖数学、物理、统计学、计算机科学等多个领域。下表按字符数统计了OpenWebMath中最常见的前十大域名：

| 域名 | 字符数 | 字符占比 |
| ------------------- | --------------- | -------- |
| stackexchange.com | 4,655,132,784 | 9.55% |
| nature.com | 1,529,935,838 | 3.14% |
| wordpress.com | 1,294,166,938 | 2.66% |
| physicsforums.com | 1,160,137,919 | 2.38% |
| github.io | 725,689,722 | 1.49% |
| zbmath.org | 620,019,503 | 1.27% |
| wikipedia.org | 618,024,754 | 1.27% |
| groundai.com | 545,214,990 | 1.12% |
| blogspot.com | 520,392,333 | 1.07% |
| mathoverflow.net | 499,102,560 | 1.02% |

# OpenWebMath 处理流程
<img src="imgs/pipeline.png" alt="OpenWebMath 处理流程概览">

OpenWebMath基于超大规模的[通用爬虫（Common Crawl）](https://commoncrawl.org/)数据集构建，该数据集包含2000亿余份HTML文档。我们对数据进行了三重筛选：仅保留（1）英语文档、（2）包含数学内容的文档，以及（3）高质量文档。相较于其他网页数据集，我们重点强化了从HTML文档中提取LaTeX内容的能力，并减少了冗余网页模板文本。

OpenWebMath的处理流程包含五个阶段：
1. **HTML文档预筛选**：
- 我们对通用爬虫（Common Crawl）中的所有HTML文档应用简易预过滤器，跳过不含数学内容的文档，避免不必要的计算开销。
2. **文本提取**：
- 从HTML文档中提取文本（含LaTeX内容），同时移除冗余网页模板文本。
3. **内容分类与筛选**：
- 采用[FastText语言识别模型](https://fasttext.cc/docs/en/language-identification.html)，仅保留英语文档。
- 使用基于[Proof-Pile](https://huggingface.co/datasets/hoskinson-center/proof-pile)训练得到的[KenLM](https://github.com/kpu/kenlm)模型，过滤高困惑度文档。
- 使用自研的_MathScore_模型过滤非数学类文档。
4. **去重**：
- 借助[text-dedup](https://github.com/ChenghaoMou/text-dedup)中的SimHash算法对数据集进行去重。
5. **人工审核**：
- 对前序步骤得到的文档进行人工审核，移除低质量页面。

如需了解处理流程的详细讨论，请参阅我们的论文。

# 许可协议
OpenWebMath采用ODC-By 1.0许可协议发布；用户同时需遵守通用爬虫（Common Crawl）的使用条款：[https://commoncrawl.org/terms-of-use/](https://commoncrawl.org/terms-of-use/)。我们未对原始数据的许可协议进行任何修改。

# 引用信息

@misc{paster2023openwebmath,
title={OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text},
author={Keiran Paster and Marco Dos Santos and Zhangir Azerbayev and Jimmy Ba},
year={2023},
eprint={2310.06786},
archivePrefix={arXiv},
primaryClass={cs.AI}
}