BAAI/IndustryCorpus_ai

该数据集是为了解决行业模型训练中数据量不足、质量低和缺乏领域专业知识的问题而构建的。通过应用22种行业数据处理操作符，从超过100TB的开源数据集中清洗和过滤出3.4TB的高质量多行业分类中英文预训练数据集。数据集包含1TB的中文数据和2.4TB的英文数据，并对中文数据进行了12种类型的标注。此外，数据集在医疗行业示范模型上进行了预训练、SFT和DPO训练，结果显示客观性能提升了20%，主观胜率为82%。数据集涵盖了18个行业类别，并提供了详细的行业分类数据大小表。

The dataset contains 3.4TB of high-quality Chinese and English multi-industry classified pre-training data, which has been cleaned and filtered through 22 industry data processing operators, resulting in 1TB of Chinese data and 2.4TB of English data. The dataset covers 18 industry categories and annotates Chinese data with 12 types of labels, including alphanumeric ratio, average line length, language confidence score, maximum line length, and perplexity. Additionally, the dataset has undergone model validation, showing a 20% performance improvement on a medical industry demonstration model.