Infinity-MM
收藏资源简介：
Infinity-MM 是一个大规模的多模态指令数据集，包含数千万个样本。通过质量过滤和去重，数据集具有高质量和多样性。数据集分为四个阶段，每个阶段包含不同类型的数据，如图像-字幕数据、通用视觉指令数据、选择性视觉指令数据和 GPT4 与合成数据。数据集的语言包括英语和中文，并且来自多个平台。数据集与 Aquila-VL-2B 模型一起使用，该模型在相同规模的模型中达到了最先进的性能。
Infinity-MM is a large-scale multimodal instruction dataset containing tens of millions of samples. Through quality filtering and deduplication, it boasts high quality and diversity. The dataset is divided into four stages, each encompassing different types of data, such as image-caption data, general visual instruction data, selective visual instruction data, as well as data related to GPT4 and synthetic data. The dataset covers both English and Chinese languages, and is sourced from multiple platforms. It is intended for use with the Aquila-VL-2B model, which achieves state-of-the-art performance among models of comparable scale.
Infinity-MM 数据集概述
数据集简介
Infinity-MM 是一个大规模的多模态指令数据集，包含数千万个样本。通过质量过滤和去重，数据集具有高质量和多样性。数据集通过开源模型和标注系统生成合成数据，使用详细的图像标注和多样的问题生成。
数据集更新
- 2024/10/24: Stage 2、Stage 3 和 Stage 4 的数据已转移，Stage 1 的数据将于 2024/11/1 完成传输。
数据集分类
| 数据阶段 | 数据类别 | 数据量 | 数据组成 |
|---|---|---|---|
| Stage 1 | 图像-标题数据 | 10M | 标题数据 10M |
| Stage 2 | 通用视觉指令数据 | 24.4M | 通用数据 7.1M, 通用OCR数据 2.6M, 文档/图表/屏幕数据 5.8M, 数学/推理数据 1.3M, 文本指令数据 7.6M |
| Stage 3 | 选择性视觉指令数据 | 6M | LLaVA-onevision 数据 3.5M, Infinity-Instruct 1.3M, Docmatix 数据 1.2M |
| Stage 4 | GPT4 & 合成数据 | 3M | GPT4 生成数据 1.7M, 合成数据 0.8M, 特定任务数据 0.4M, Infinity-Preference 数据 0.1M |
数据格式示例
json { "id": "10", "source": "llava", "conversations": [ { "from": "human", "value": "<image> Can you create a descriptive poem about the scene?" }, { "from": "gpt", "value": "Upon a plate of verdant hue, A cluster of roasted nuts we view. Cashews golden, shining bright, Almonds dark, a roasted delight. Sprinkled seeds of sesame light, Enhancing flavors, a gentle bite. On white wood they rest in grace, A rustic charm, a peaceful place. Simple joys in a vibrant array, A perfect treat to start the day." } ], "image": "/path/of/the/image", "ram++_tags": ["wall", "dry", "grassy", "hill", "stone", "sun", "sunset"], "ram++_tags_score": [9.56411075592041, 2.3733813762664795, 1.4329272508621216, 1.9840935468673706, 1.9766467809677124, 2.255882501602173, 2.575751781463623], "phash": [12512305226191801180], "qw2vl_loss": 3.0559005737304688 }
数据字段解释
- id: 记录的ID。
- source: 记录的来源。
- conversations: 记录的对话内容。
- image: 图像的绝对路径。
- ram++_tags & ram++_tags_score: 由 Ram++ 模型生成的图像标签及其分数。
- phash: 图像的phash值。
- qw2vl_loss: 由 Qwen2-VL-2B 模型计算的值。
数据来源
| 数据来源 | 数据量 |
|---|---|
| Emu2 | 10M |
| LVIS-Instruct | 223K |
| LLaVA-CC3M-Pretrain-595K | 595K |
| Visdial | 116K |
| Sharegpt4 | 3.2M |
| STVQA | 43K |
| MMC-INST | 500K |
| MathV360K | 338K |
| MMC-Alignment | 250K |
| DocReason | 26K |
| ALLaVA | 1.7M |
| Cocotext | 163K |
| Docvqa | 16K |
| Geoqa+ | 72K |
| DocDownstream | 700K |
| Cambrian | 8.3M |
| DocStruct4M | 4M |
| LLaVA-onevision | 4M |
| Docmatix | 1.2M |
| Infinity-Instruct | 7M |
| 合成数据 | 0.8M |
引用
如果该数据集对您的研究有帮助，请引用以下工作：
@misc{gu2024infinitymmscalingmultimodalperformance, title={Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data}, author={Shuhao Gu and Jialing Zhang and Siyuan Zhou and Kevin Yu and Zhaohu Xing and Liangdong Wang and Zhou Cao and Jintao Jia and Zhuoyi Zhang and Yixuan Wang and Zhenchong Hu and Bo-Wen Zhang and Jijie Li and Dong Liang and Yingli Zhao and Yulong Ao and Yaoqi Liu and Fangxiang Feng and Guang Liu}, year={2024}, eprint={2410.18558}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2410.18558}, }
