WanJuan1.0（书生·万卷）

书生·万卷1.0为书生·万卷多模态语料库的首个开源版本，包含文本数据集、图文数据集、视频数据集三部分，数据总量超过2TB。 目前，书生·万卷1.0已被应用于书生·多模态、书生·浦语的训练。通过对高质量语料的“消化”，书生系列模型在语义理解、知识问答、视觉理解、视觉问答等各类生成式任务表现出的优异性能。 （邮箱联系：[e-mail]）Shusheng·Wanjuan 1.0 is the first open-source version of the Shusheng·Wanjuan multimodal corpus, which consists of three subsets: text datasets, image-text datasets, and video datasets, with a total data volume exceeding 2 TB. Currently, Shusheng·Wanjuan 1.0 has been applied to the training of Shusheng·Multimodal and Shusheng·Puyu. By leveraging high-quality corpus resources, Shusheng-series models demonstrate excellent performance across various generative tasks including semantic understanding, knowledge question answering, visual understanding, and visual question answering. Contact: [e-mail]

搜集汇总

背景与挑战

背景概述

WanJuan1.0（书生·万卷）是一个大型多模态语料库，包含文本、图文和视频数据，总量超过2TB，覆盖科技、文学等多个领域。数据集经过精细清洗、去重和价值对齐处理，确保高质量和安全可信，采用统一格式便于使用，已成功应用于书生系列模型的训练，显著提升了模型在语义理解和生成任务上的性能。