OpenWebMath 开放网络数学训练数据集

OpenWebMath 是一个包含来自互联网的大部分高质量数学文本的数据集。它是从 Common Crawl 上超过 200B 个 HTML 文件中过滤和提取的，最终形成一组 630 万个文档，总共包含 14.7B 个 tokens 。 OpenWebMath 旨在用于预训练和微调大型语言模型。

OpenWebMath is a dataset containing most high-quality mathematical texts sourced from the Internet. It was filtered and extracted from over 200 billion HTML files on Common Crawl, ultimately forming a collection of 6.3 million documents with a total of 14.7 billion tokens. OpenWebMath is designed for pre-training and fine-tuning large language models.

搜集汇总

背景与挑战

背景概述

OpenWebMath是一个从Common Crawl中提取的高质量数学文本数据集，包含630万个文档和14.7B个tokens，适用于预训练和微调大型语言模型。该数据集大小为44.21 GB，发布于2年前。