wchai/AuroraCap-trainset

AuroraCap Trainset是一个用于视频详细描述任务的数据集，包含超过2000万高质量的图像/视频-文本对。数据集的训练过程分为三个阶段：预训练阶段、视觉阶段和语言阶段。在预训练阶段，视觉特征与大型语言模型的词嵌入空间对齐；在视觉阶段，解冻预训练的视觉Transformer（ViT）并进行训练以获得更好的泛化能力；在语言阶段，进行端到端的训练，所有组件均可训练。数据集支持英语和中文，大小在10M到100M之间。

The dataset contains over 20 million high-quality image/video-text pairs used to train the AuroraCap model. The training process is divided into three stages: pretraining stage, vision stage, and language stage. The pretraining stage aligns visual features with the language model; the vision stage trains with public data to improve generalization; the language stage conducts end-to-end training where all components are trainable.