BAAI/IndustryCorpus_emotion

该数据集是一个高质量的多行业分类中英文预训练数据集，包含1TB的中文数据和2.4TB的英文数据。数据集从超过100TB的开源数据集中筛选出来，并应用了22种行业数据处理操作符进行清洗和过滤。中文数据被标注了12种标签，包括字母数字比例、平均行长度、语言置信度分数、最大行长度和困惑度等。数据集在医疗行业示范模型上的性能验证结果显示，客观性能提升了20%，主观胜率为82%。数据集涵盖了18个行业分类，包括医疗、教育、文学、金融、旅游、法律、体育、汽车、新闻等。数据集还进行了规则过滤、模型过滤和数据去重等处理。当前提供的是情感行业的子数据集。

This dataset is a high-quality multi-industry classified Chinese and English pre-training dataset, containing 1TB of Chinese data and 2.4TB of English data. The dataset was filtered from over 100TB of open-source datasets and cleaned and filtered using 22 industry data processing operators. The Chinese data is annotated with 12 types of labels, including alphanumeric ratio, average line length, language confidence score, maximum line length, and perplexity. The datasets performance was validated on a medical industry demonstration model, showing a 20% improvement in objective performance and a subjective win rate of 82%. The dataset covers 18 industry categories, including medical, education, literature, finance, travel, law, sports, automotive, news, etc. The dataset also underwent rule-based filtering, model-based filtering, and data deduplication. The current provided sub-dataset is for the emotion industry.