chinese-fineweb-edu

Chinese Fineweb Edu数据集是一个精心构建的高质量中文预训练语料数据集，专为教育领域的自然语言处理任务设计。该数据集通过严格的筛选和去重流程，利用少量数据训练打分模型进行评估，从海量的原始数据中提取出高价值的教育相关内容，确保数据的质量和多样性。最终，数据集包含约90M条高质量的中文文本数据，总大小约为300GB。

Chinese Fineweb Edu Dataset is a meticulously constructed high-quality Chinese pre-training corpus dataset designed specifically for natural language processing tasks in the education domain. This dataset undergoes strict filtering and deduplication processes, uses a small amount of training data to train a scoring model for evaluation, extracts high-value education-related content from a massive volume of raw data, and ensures the quality and diversity of the dataset. Ultimately, the dataset contains approximately 90 million high-quality Chinese text entries, with a total size of around 300 GB.

搜集汇总

背景与挑战

背景概述

Chinese Fineweb Edu是一个高质量的中文教育领域预训练语料数据集，包含约90M条经过严格筛选的文本数据，总大小约300GB。该数据集通过专业评分模型和去重处理，确保内容的教育价值和多样性，适用于教育相关的自然语言处理任务。