BAAI/IndustryCorpus_news

该数据集是通过对超过100TB的开源数据集（包括WuDaoCorpora、BAAI-CCI、redpajama和SkyPile-150B）进行清洗和过滤，构建了3.4TB的高质量多行业分类中英文预训练数据集。数据集包括1TB的中文数据和2.4TB的英文数据，并对中文数据进行了12种类型的标注。数据集在医疗行业示范模型上进行了预训练、SFT和DPO训练，结果显示在客观性能上有20%的提升，主观胜率为82%。数据集涵盖了18个行业类别，包括医疗、教育、文学、金融等，并提供了各行业的数据大小。

This dataset was constructed by cleaning and filtering over 100TB of open-source datasets, including WuDaoCorpora, BAAI-CCI, redpajama, and SkyPile-150B, resulting in a 3.4TB high-quality multi-industry classified Chinese and English pre-training dataset. The dataset includes 1TB of Chinese data and 2.4TB of English data, with 12 types of labels annotated for the Chinese data. The dataset was used for continued pre-training, SFT, and DPO training on a medical industry demonstration model, showing a 20% improvement in objective performance and an 82% subjective win rate. The dataset covers 18 industry categories, including medical, education, literature, finance, etc., and provides data sizes for each industry.