BAAI/IndustryCorpus_medicine

该数据集是一个高质量的多行业分类中文和英文预训练数据集，总数据量为3.4TB，其中中文数据1TB，英文数据2.4TB。数据集通过22个行业数据处理操作符从超过100TB的开源数据集中筛选出来，涵盖了18个行业类别，包括医疗、教育、文学、金融、旅游、法律、体育、汽车、新闻等。中文数据被标注了12种标签，如字母数字比例、平均行长度、语言置信度分数等。数据集的应用效果通过医疗行业示范模型的持续预训练、SFT和DPO训练进行了验证，结果显示客观性能提升了20%，主观胜率达到82%。

This dataset is a high-quality multi-industry classified Chinese and English pre-training dataset with a total data volume of 3.4TB, including 1TB of Chinese data and 2.4TB of English data. The dataset was filtered from over 100TB of open-source datasets using 22 industry data processing operators, covering 18 industry categories, including medical, education, literature, finance, travel, law, sports, automotive, news, etc. The Chinese data is annotated with 12 types of labels, such as alphanumeric ratio, average line length, language confidence score, etc. The datasets performance was validated through continued pre-training, SFT, and DPO training on a medical industry demonstration model, showing a 20% improvement in objective performance and a subjective win rate of 82%.