prithivMLmods/OpenWeb1M

OpenWeb数据集网络集合是从FineWeb数据集派生出来的，包含超过15万亿个清理过和去重的英文网络数据标记，这些数据来自CommonCrawl。数据处理流水线针对LLM性能进行了优化，并且从Hugging Face的FineWeb集合中提取了必要的数据集。这个数据集是通过处理96个CommonCrawl转存，包括从2013年夏天到2024年4月的网络数据创建的。FineWeb包含了各种英文领域的主题，主要旨在作为大型语言模型预训练数据集的公共数据研究工具。CommonCrawl数据经过仔细的处理、过滤和去重，使用Datatrove库，形成了大约15万亿个标记的最大公开清洁LLM预训练数据集（使用GPT-2标记器）。

The OpenWeb Datasets Web Collection, derived from the FineWeb dataset, consists of more than 15 trillion tokens of cleaned and deduplicated English web data from CommonCrawl. The data processing pipeline is optimized for LLM performance, and the necessary set of datasets has been extracted from Hugging Faces FineWeb collections. This dataset was created by processing 96 CommonCrawl dumps, comprising web data crawled from the summer of 2013 to April 2024. FineWeb includes a variety of domains and topics in English and is primarily intended to serve as a research artifact for public data in the context of pretraining datasets for large language models. The CommonCrawl data was carefully processed, filtered, and deduplicated using the Datatrove library, resulting in the largest publicly available clean LLM pretraining dataset, containing approximately 15 trillion tokens (using the GPT-2 tokenizer).