BAAI/IndustryCorpus_mathematics

该数据集是一个多行业分类的中英文预训练数据集，通过22个行业数据处理操作符从超过100TB的开源数据集中清洗和过滤得到，包含1TB的中文数据和2.4TB的英文数据。数据集涵盖了18个行业类别，如医疗、教育、文学、金融等。中文数据被标注了12种标签，包括字母数字比例、平均行长度、语言置信度分数等。数据集还进行了模型性能验证，结果显示在医疗行业示范模型上，客观性能提高了20%，主观胜率为82%。数据集提供了基于规则和模型的过滤方法，以及数据去重处理。

This dataset is a multi-industry classified Chinese and English pre-training dataset, cleaned and filtered from over 100TB of open-source datasets using 22 industry data processing operators, containing 1TB of Chinese data and 2.4TB of English data. The dataset covers 18 industry categories, such as medical, education, literature, finance, etc. The Chinese data is annotated with 12 types of labels, including alphanumeric ratio, average line length, language confidence score, etc. The dataset also underwent model performance validation, showing a 20% improvement in objective performance and an 82% subjective win rate on a medical industry demonstration model. The dataset provides rule-based and model-based filtering methods, as well as data deduplication processing.