BAAI/IndustryCorpus_law

该数据集是通过对超过100TB的开源数据集（包括WuDaoCorpora、BAAI-CCI、redpajama和SkyPile-150B等）进行清洗和过滤，生成了3.4TB的高质量多行业分类的中英文预训练数据集。其中，中文数据1TB，英文数据2.4TB。中文数据标注了12种标签，包括字母数字比例、平均行长度、语言置信度分数、最大行长度和困惑度等。数据集的应用效果通过在一个医疗行业示范模型上的训练得到了验证，结果显示客观性能提升了20%，主观胜率为82%。数据集包含18个行业类别，如医疗、教育、文学、金融、旅游、法律、体育、汽车、新闻等。每个类别的数据量也有所不同，例如法律类数据274.6GB，教育类数据458.1GB等。

This dataset was constructed by cleaning and filtering over 100TB of open-source datasets, including WuDaoCorpora, BAAI-CCI, redpajama, and SkyPile-150B, resulting in 3.4TB of high-quality multi-industry classified Chinese and English pre-training datasets. The filtered data consists of 1TB of Chinese data and 2.4TB of English data. The Chinese data is annotated with 12 types of labels, including alphanumeric ratio, average line length, language confidence score, maximum line length, and perplexity. The datasets performance was validated through continued pre-training, SFT, and DPO training on a medical industry demonstration model, showing a 20% improvement in objective performance and a subjective win rate of 82%. The dataset covers 18 industry categories, such as medical, education, literature, finance, travel, law, sports, automotive, and news. The data size for each category varies, for example, 274.6GB for law, 458.1GB for education, etc.