OpenWeb1M
收藏魔搭社区2025-10-09 更新2025-02-08 收录
下载链接：
https://modelscope.cn/datasets/prithivMLmods/OpenWeb1M
下载链接
链接失效反馈官方服务：
资源简介：
# **OpenWeb Datasets Web Collection**
The OpenWeb Datasets Web Collection, derived from the 'FineWeb' dataset, consists of more than 15 trillion tokens of cleaned and deduplicated English web data from CommonCrawl. The data processing pipeline is optimized for LLM performance, and the necessary set of datasets has been extracted from Hugging Face's FineWeb collections. This dataset was created by processing 96 CommonCrawl dumps, comprising web data crawled from the summer of 2013 to April 2024. FineWeb includes a variety of domains and topics in English and is primarily intended to serve as a research artifact for public data in the context of pretraining datasets for large language models. The CommonCrawl data was carefully processed, filtered, and deduplicated using the Datatrove library, resulting in the largest publicly available clean LLM pretraining dataset, containing approximately 15 trillion tokens (using the GPT-2 tokenizer).
## FineWeb Dataset Overview
| **Dataset Name** | **Total Entries** | **Dataset Link** |
|-----------------|-----------------|-----------------|
| FineWeb | 25B | [FineWeb Dataset](https://huggingface.co/datasets/HuggingFaceFW/fineweb) |
# **OpenWeb数据集网页合集**
本OpenWeb数据集网页合集源自FineWeb数据集，包含源自CommonCrawl的经清洗、去重的英文网页数据，总量超15万亿Token（Token）。该数据集的处理管线针对大语言模型（LLM/Large Language Model）的性能进行了优化，所需数据集子集均从Hugging Face的FineWeb数据集中提取而来。本数据集通过处理96份CommonCrawl数据转储文件构建而成，涵盖2013年夏季至2024年4月期间爬取的网页数据。FineWeb包含多领域、多主题的英文数据，其核心定位为面向大语言模型预训练数据集场景的公开数据研究工具集。本次采集的CommonCrawl数据通过Datatrove库完成了精细化处理、过滤与去重操作，最终形成目前规模最大的公开可获取的清洗版大语言模型预训练数据集，采用GPT-2分词器计算的总Token量约为15万亿。
## FineWeb数据集概览
| **数据集名称** | **总条目数** | **数据集链接** |
|-----------------|-----------------|-----------------|
| FineWeb | 250亿 | [FineWeb数据集](https://huggingface.co/datasets/HuggingFaceFW/fineweb) |
提供机构：
maas
创建时间：
2025-02-07
