BAAI/IndustryCorpus_travel

该数据集是一个高质量的多行业分类中文和英文预训练数据集，总数据量为3.4TB，其中中文数据1TB，英文数据2.4TB。数据集通过22个行业数据处理操作符从超过100TB的开源数据集中筛选出来，涵盖了18个行业类别，如医疗、教育、文学、金融、旅游、法律、体育、汽车、新闻等。中文数据还标注了12种标签，包括字母数字比例、平均行长度、语言置信度分数等。数据集的应用验证表明，其在医疗行业模型上的表现有显著提升。

This dataset is a high-quality multi-industry classified Chinese and English pre-training dataset with a total data volume of 3.4TB, including 1TB of Chinese data and 2.4TB of English data. The dataset was filtered from over 100TB of open-source datasets using 22 industry data processing operators, covering 18 industry categories such as medical, education, literature, finance, travel, law, sports, automotive, news, etc. The Chinese data is also annotated with 12 types of labels, including alphanumeric ratio, average line length, language confidence score, etc. The application validation of the dataset shows significant performance improvement in the medical industry model.