IndustryCorpus_film

[[中文主页]](README_ZH.md)

Industry models play a crucial role in driving enterprise intelligence transformation and innovative development. High-quality industry data is key to improving the performance of large models and realizing industry applications. However, datasets currently used for industry model training generally suffer from issues such as insufficient data volume, low quality, and lack of domain expertise.

To address these problems, we constructed and applied 22 industry data processing operators to clean and filter 3.4TB of high-quality multi-industry classified Chinese and English language pre-training datasets from over 100TB of open-source datasets including WuDaoCorpora, BAAI-CCI, redpajama, and SkyPile-150B. The filtered data consists of 1TB of Chinese data and 2.4TB of English data. To facilitate user utilization, we annotated the Chinese data with 12 types of labels including alphanumeric ratio, average line length, language confidence score, maximum line length, and perplexity.

Furthermore, to validate the dataset's performance, we conducted continued pre-training, SFT, and DPO training on a medical industry demonstration model. The results showed a 20% improvement in objective performance and a subjective win rate of 82%.

Industry categories: 18 categories including medical, education, literature, finance, travel, law, sports, automotive, news, etc.
Rule-based filtering: Traditional Chinese conversion, email removal, IP address removal, link removal, Unicode repair, etc.
Chinese data labels: Alphanumeric ratio, average line length, language confidence score, maximum line length, perplexity, toxicity character ratio, etc.
Model-based filtering: Industry classification language model with 80% accuracy
Data deduplication: MinHash document-level deduplication
Data size: 1TB Chinese, 2.4TB English

Industry classification data size:

| Industry Category | Data Size (GB) | Industry Category | Data Size (GB) |
| :-------------------:|:----------------:|:-------------------:|:----------------:|
| Programming | 4.1 | Politics | 326.4 |
| Law | 274.6 | Mathematics | 5.9 |
| Education | 458.1 | Sports | 442 |
| Finance | 197.8 | Literature | 179.3 |
| Computer Science | 46.9 | News | 564.1 |
| Technology | 333.6 | Film & TV | 162.1 |
| Travel | 82.5 | Medicine | 189.4 |
| Agriculture | 41.6 | Automotive | 40.8 |
| Emotion | 31.7 | Artificial Intelligence | 5.6 |
| Total (GB) | 3386.5 | | |

For the convenience of users to download and use, we have split the large dataset into sub-datasets for 18 industries. The current one is the sub-dataset for the film industry.

Data processing workflow:


[[中文主页]](README_ZH.md)

行业模型对于推动企业智能化转型与创新发展具有至关重要的作用。高质量的行业数据是提升大语言模型（Large Language Model, LLM）性能、实现行业落地应用的核心关键。但当前用于行业模型训练的数据集普遍存在数据体量不足、质量欠佳、缺乏领域专业知识等痛点。

为解决上述问题，我们构建并应用了22个行业数据处理算子，从悟道语料库（WuDaoCorpora）、BAAI-CCI、RedPajama、SkyPile-150B等超过100TB的开源数据集中，清洗筛选出3.4TB的高质量多行业分类中英双语预训练数据集。经筛选后的数据包含1TB中文数据与2.4TB英文数据。为便于用户使用，我们为中文数据标注了12类标签，涵盖字母数字占比、平均行长度、语言置信度得分、最大行长度以及困惑度（perplexity）等。

此外，为验证该数据集的实际性能，我们在医疗行业演示模型上开展了持续预训练、监督微调（Supervised Fine-Tuning, SFT）与直接偏好优化（Direct Preference Optimization, DPO）训练。实验结果显示，模型客观性能提升20%，主观胜率达82%。

行业分类：包含医疗、教育、文学、金融、旅游、法律、体育、汽车、新闻等共18个类别。
基于规则的过滤流程：繁体中文转换、邮件移除、IP地址清除、链接去除、Unicode修复等。
中文数据标签：字母数字占比、平均行长度、语言置信度得分、最大行长度、困惑度、有害字符占比等。
基于模型的过滤：采用准确率达80%的行业分类语言模型。
数据去重：通过MinHash实现文档级去重。
数据规模：中文数据1TB，英文数据2.4TB。

行业分类数据规模：

| 行业分类 | 数据规模（GB） | 行业分类 | 数据规模（GB） |
|:-------------------:|:----------------:|:-------------------:|:----------------:|
| 编程 | 4.1 | 政治 | 326.4 |
| 法律 | 274.6 | 数学 | 5.9 |
| 教育 | 458.1 | 体育 | 442 |
| 金融 | 197.8 | 文学 | 179.3 |
| 计算机科学 | 46.9 | 新闻 | 564.1 |
| 科技 | 333.6 | 影视 | 162.1 |
| 旅游 | 82.5 | 医疗 | 189.4 |
| 农业 | 41.6 | 汽车 | 40.8 |
| 情感 | 31.7 | 人工智能 | 5.6 |
| 总计（GB） | 3386.5 | | |

为方便用户下载与使用，我们将该大型数据集拆分为18个行业的子数据集，当前提供的为影视行业子数据集。

数据处理流程：

搜集汇总

背景与挑战

背景概述

IndustryCorpus_film是IndustryCorpus的一个子数据集，专注于电影行业，旨在解决行业模型训练中数据不足和质量问题。它通过22个数据处理操作符从多源开源数据中清洗和过滤出高质量中英文预训练数据，并标注了多种标签以方便使用。该数据集是整体数据的一部分，其中中文数据1TB、英文数据2.4TB，经行业分类和去重等处理。