Data-Juicer

该数据集包含了8000亿个标记，用于对早期退出的大型语言模型（LLMs）进行预训练。该数据集不仅服务于不同规模早期退出LLMs的训练，而且对训练和推理过程的效率提升有着重要贡献。其规模达到了800亿个标记，任务是对大型语言模型进行预训练。

This dataset contains 800 billion tokens, specifically designed for pre-training early-exit Large Language Models (LLMs). It not only supports the training of early-exit LLMs across different scales, but also makes substantial contributions to enhancing the efficiency of both training and inference workflows. Boasting a scale of 800 billion tokens, its core task is the pre-training of large language models.