BAAI/IndustryCorpus

该数据集是一个高质量的多行业分类中英文预训练数据集，通过22个行业数据处理操作符从超过100TB的开源数据集中清洗和过滤出3.4TB的高质量数据，包括1TB中文数据和2.4TB英文数据。中文数据标注了12种标签，如字母数字比例、平均行长度、语言置信度分数等。数据集在医疗行业示范模型上的持续预训练、SFT和DPO训练中表现出色，客观性能提升了20%，主观胜率达到82%。数据集涵盖18个行业类别，如医疗、教育、文学、金融等，并进行了规则和模型基础的过滤、数据去重等处理。

This dataset consists of 3.4TB of high-quality multi-industry classified Chinese and English pre-training data, filtered from over 100TB of open-source datasets using 22 industry data processing operators. It includes 1TB of Chinese data and 2.4TB of English data, with 12 types of labels annotated for the Chinese data. The dataset covers 18 industry categories such as medical, education, literature, finance, and uses rule-based and model-based filtering methods, as well as document-level deduplication techniques. Performance validation on a medical industry demonstration model showed a 20% improvement in objective performance and an 82% subjective win rate.

搜集汇总

背景与挑战

背景概述

BAAI/IndustryCorpus是一个多语言的行业语料库，包含农业等18个行业的高质量文本数据。数据集大小为100M到1B，格式为json，适用于多语言和跨行业的自然语言处理任务。数据经过清洗和标注，支持行业模型训练和应用开发。