BAAI/IndustryCorpus_sports

该数据集是通过对超过100TB的开源数据集进行清洗和过滤得到的，最终生成了3.4TB的高质量多行业分类中英文预训练数据集。数据集包含1TB的中文数据和2.4TB的英文数据，并且对中文数据进行了12种类型的标注。数据集在医疗行业示范模型上的应用表现出了显著的性能提升。数据集涵盖了18个行业类别，包括医疗、教育、文学、金融、旅游、法律、体育、汽车、新闻等。为了方便用户使用，数据集被分割为18个行业的子数据集，当前提供的是体育行业的子数据集。

The dataset consists of 3.4TB of high-quality multi-industry classified Chinese and English pre-training data, which has been cleaned and filtered through 22 industry data processing operators, resulting in 1TB of Chinese data and 2.4TB of English data. The dataset covers 18 industry categories and includes 12 types of labels for Chinese data, such as alphanumeric ratio, average line length, language confidence score, maximum line length, and perplexity. The performance of the dataset was validated through continued pre-training, SFT, and DPO training on a medical industry demonstration model, showing a 20% improvement in objective performance and an 82% subjective win rate.