BAAI/IndustryCorpus_film

该数据集是一个高质量的多行业分类中英文预训练数据集，通过22个行业数据处理操作符从超过100TB的开源数据集中清洗和过滤出3.4TB的高质量数据，其中包含1TB的中文数据和2.4TB的英文数据。中文数据被标注了12种标签，如字母数字比例、平均行长度、语言置信度分数等。数据集在医疗行业示范模型上的持续预训练、SFT和DPO训练中表现出色，客观性能提升了20%，主观胜率达到82%。数据集涵盖了18个行业类别，包括医疗、教育、文学、金融等，并进行了基于规则和模型的过滤、数据去重等处理。

This dataset contains 3.4TB of high-quality multi-industry classified Chinese and English pre-training data, which has been cleaned and filtered through 22 industry data processing operators, resulting in 1TB of Chinese data and 2.4TB of English data. The dataset covers 18 industry categories and includes 12 types of labels for Chinese data, such as alphanumeric ratio, average line length, language confidence score, maximum line length, and perplexity. The performance of the dataset was validated through continued pre-training, SFT, and DPO training on a medical industry demonstration model, showing a 20% improvement in objective performance and an 82% subjective win rate.