opencsg/chinese-fineweb-edu-v2

Chinese Fineweb Edu v2 是Chinese Fineweb Edu的全新升级版，专为教育领域的自然语言处理（NLP）任务设计和优化的高质量中文预训练数据集。该数据集在前一版本的基础上进行了大规模的改进和扩展，致力于为研究人员和开发者提供更加多样化、广泛适用的教育类语料资源。Fineweb Edu v2 不仅数据量达到188M条数据，约420B tokens，还优化了数据的筛选方式和打分模型，以确保其在教育领域的有效性和实用性。

The Chinese Fineweb Edu Dataset V2 is a comprehensive upgrade of the original Chinese Fineweb Edu, designed and optimized for natural language processing (NLP) tasks in the education sector. This high-quality Chinese pretraining dataset has undergone significant improvements and expansions, with a dataset size of 188 million entries (approximately 420 billion tokens). The dataset has optimized data filtering methods and scoring models, utilizing the OpenCSG csg-wukong-enterprise V2 model, which excels in text comprehension and processing. The construction process of the dataset includes meticulous optimization of the prompt design for data filtering, ensuring the selection of text with educational value and practicality. The data sources have also been further extended, covering multiple high-quality datasets from various fields and sources, enhancing the diversity and applicability of the dataset.