Fineweb-Edu-Chinese-V2.1

Chinese Fineweb Edu Dataset V2.1是专为教育领域的自然语言处理任务设计的增强版本。它包含map-cc和opencsg-cc两个新的数据来源，并根据质量评分对数据进行组织，总规模达到2.27TB，包含约1.5万亿tokens。该数据集支持灵活选择不同质量评分范围的数据进行训练，适用于文本生成等任务，并采用Apache 2.0和OpenCSG社区许可证授权，商业用途需邮件申请。

Chinese Fineweb Edu Dataset V2.1 is an enhanced version specifically designed for natural language processing tasks in the educational domain. It includes two new data sources: map-cc and opencsg-cc, organizes its data based on quality scores, with a total scale of 2.27 TB and approximately 1.5 trillion tokens. This dataset supports flexible selection of data within different quality score ranges for training, is applicable to tasks such as text generation, and is licensed under Apache 2.0 and the OpenCSG Community License. Commercial usage requires prior email application.

搜集汇总

背景与挑战

背景概述

Chinese Fineweb Edu Dataset V2.1是一个专为教育领域自然语言处理任务设计的高质量数据集，包含两个新数据源（map-cc和opencsg-cc），总规模2.27TB（约1.5万亿tokens），数据按质量评分组织，支持灵活选择训练数据。采用Apache 2.0和OpenCSG社区许可证授权，商业用途需申请许可。