BAAI/IndustryCorpus_programming

该数据集是一个高质量的多行业分类中文和英文预训练数据集，总数据量超过1TB，其中中文数据1TB，英文数据2.4TB。数据集通过22种行业数据处理操作符从超过100TB的开源数据集中清洗和过滤出，并标注了12种标签，如字母数字比例、平均行长度、语言置信度分数等。数据集在医疗行业示范模型上进行了预训练、SFT和DPO训练，结果显示客观性能提升了20%，主观胜率为82%。数据集包含18个行业类别，如医疗、教育、文学、金融等，并提供了每个行业类别的数据大小。

The dataset contains 3.4TB of high-quality multi-industry classified Chinese and English pre-training data, which has been cleaned and filtered through 22 industry data processing operators, resulting in 1TB of Chinese data and 2.4TB of English data. The dataset covers 18 industry categories and annotates Chinese data with 12 types of labels, including alphanumeric ratio, average line length, language confidence score, maximum line length, and perplexity. Additionally, the dataset has undergone model validation, showing performance improvements on a medical industry demonstration model, with a 20% increase in objective performance and an 82% subjective win rate.