BAAI/IndustryCorpus_literature

该数据集是一个高质量的多行业分类中文和英文预训练数据集，总规模为3.4TB，其中中文数据1TB，英文数据2.4TB。数据集从超过100TB的开源数据集中筛选而来，使用了22种行业数据处理算子进行清洗和过滤。数据集包含18个行业分类，如医疗、教育、文学、金融等，每个行业的数据规模不同。中文数据还标注了12种标签，如字母数字比例、平均行长度、语言置信度分数等。数据集经过基于规则和基于模型的过滤，确保数据质量，并进行了文档级去重。数据集的性能验证结果显示，在医疗行业示范模型上的训练效果显著提升，客观性能提高了20%，主观胜率达到82%。

This dataset contains 3.4TB of high-quality multi-industry classified Chinese and English pre-training data, which has been cleaned and filtered through 22 industry data processing operators, resulting in 1TB of Chinese data and 2.4TB of English data. The dataset covers 18 industry categories and includes 12 types of labels for Chinese data, such as alphanumeric ratio, average line length, language confidence score, maximum line length, and perplexity. The performance of the dataset was validated through continued pre-training, SFT, and DPO training on a medical industry demonstration model, showing a 20% improvement in objective performance and a subjective win rate of 82%.