Fineweb-Edu-Chinese-V2.1

Chinese Fineweb Edu Dataset V2.1是专为教育领域的自然语言处理任务设计的增强版本。它包含map-cc和opencsg-cc两个新的数据来源，并根据质量评分对数据进行组织，总规模达到2.27TB，包含约1.5万亿tokens。该数据集支持灵活选择不同质量评分范围的数据进行训练，适用于文本生成等任务，并采用Apache 2.0和OpenCSG社区许可证授权，商业用途需邮件申请。

The Chinese Fineweb Edu Dataset V2.1 is an enhanced version specifically designed for natural language processing tasks in the educational domain. It includes two new data sources: map-cc and opencsg-cc, and organizes data based on quality scores. The total size of the dataset reaches 2.27 TB, containing approximately 1.5 trillion tokens. It supports flexible selection of data within different quality score ranges for training, and is applicable to tasks such as text generation. The dataset is licensed under Apache 2.0 and the OpenCSG Community License, and commercial use requires prior email application.

搜集汇总

背景与挑战

背景概述

Chinese Fineweb Edu Dataset V2.1是专为教育领域自然语言处理任务优化的增强数据集，引入了map-cc和opencsg-cc两个新数据源，总规模达2.27TB（约1.5万亿tokens）。数据按质量评分（2-5分）组织，支持灵活选择不同评分范围进行训练，适用于文本生成等任务，采用Apache 2.0和OpenCSG社区许可证授权。