MMInstruction/M3IT
收藏Hugging Face2023-11-24 更新2024-03-04 收录
下载链接：
https://hf-mirror.com/datasets/MMInstruction/M3IT
下载链接
链接失效反馈官方服务：
资源简介：
---
license: other
task_categories:
- image-to-text
- image-classification
size_categories:
- 1M<n<10M
language:
- en
- zh
---
# Dataset Card for M3IT
Project Page: [M3IT](https://m3-it.github.io/)
## Dataset Description
- **Homepage: https://huggingface.co/datasets/MMInstruction/M3IT**
- **Repository: https://huggingface.co/datasets/MMInstruction/M3IT**
- **Paper:**
- **Leaderboard:**
- **Point of Contact:**
### Languages
English and Chinese. 80 translated version can be found at [M3IT-80](https://huggingface.co/datasets/MMInstruction/M3IT-80).
## Dataset Statistics
Our dataset compiles diverse tasks of classical vision-language tasks, including captioning,
visual question answering~(VQA), visual conditioned generation, reasoning and classification.
### Instruction Statistics
| Task | #Instructions |
|---------------------------|---------------|
| Image Captioning | 52 |
| Classification | 113 |
| Visual Question Answering | 95 |
| Knowledgeable Visual QA | 40 |
| Reasoning | 60 |
| Generation | 40 |
| Total | 400 |
### Task Statistics
| Task | Description | #Train | #Val | #Test |
|---------------------------|-----------------------------------------------------------------|---------|---------|---------|
| Image Captioning | Given an image, write a description for the image. | 679,087 | 41,462 | 27,499 |
| Classification | Given an image, classify the image into pre-defined categories. | 238,303 | 100,069 | 21,206 |
| Visual Question Answering | Given an image, answer a question relevant to the image. | 177,633 | 46,314 | 10,828 |
| Knowledgeable Visual QA | Given an image, answer the question requires outside knowledge. | 39,981 | 11,682 | 5,477 |
| Reasoning | Given an image, conduct reasoning over the images. | 99,372 | 11,500 | 10,000 |
| Generation | Given an image, make compositions with certain requirements. | 145,000 | 11,315 | 17,350 |
| Chinese | CAP, CLS, VQA, and GEN tasks in Chinese. | 192,076 | 77,306 | 4,100 |
| Video | CAP, CLS, and VQA tasks on video-language datasets. | 20,868 | 7,542 | 9,294 |
| Multi-lingual | Translated tasks in 80 languages | 0 | 240,000 | 184,000 |
### Detailed Dataset Statistics
| Task | Dataset | #Train | #Val | #Test |
|---------------------------|------------------------------|---------|--------|--------|
| Image Captioning | `coco` | 566,747 | 25,010 | 25,010 |
| | `textcap` | 97,765 | 13,965 | 0 |
| | `image-paragraph-captioning` | 14,575 | 2,487 | 2,489 |
| Classification | `coco-goi` | 30,000 | 2,000 | 0 |
| | `coco-text` | 118,312 | 27,550 | 0 |
| | `imagenet` | 30,000 | 50,000 | 0 |
| | `coco-itm` | 30,000 | 5,000 | 5,000 |
| | `snli-ve` | 20,000 | 14,339 | 14,740 |
| | `mocheg` | 4,991 | 180 | 466 |
| | `iqa` | 5,000 | 1,000 | 1,000 |
| Visual Question Answering | `vqa-v2` | 30,000 | 30,000 | 0 |
| | `shapes` | 13,568 | 1,024 | 1,024 |
| | `docvqa` | 39,463 | 5,349 | 0 |
| | `ocr-vqa` | 11,414 | 4,940 | 0 |
| | `st-vqa` | 26,074 | 0 | 4,070 |
| | `text-vqa` | 27,113 | 0 | 5,734 |
| | `gqa` | 30,001 | 5,001 | 0 |
| Knowledgeable Visual QA | `okvqa` | 9,009 | 5,046 | 0 |
| | `a-okvqa` | 17,056 | 1,145 | 0 |
| | `science-qa` | 12,726 | 4,241 | 4,241 |
| | `viquae` | 1,190 | 1,250 | 1,236 |
| Reasoning | `clevr` | 30,000 | 2,000 | 0 |
| | `nlvr` | 29,372 | 2,000 | 0 |
| | `vcr` | 25,000 | 5,000 | 5,000 |
| | `visual-mrc` | 15,000 | 2,500 | 5,000 |
| | `winoground` | 0 | 0 | 800 |
| Generation | `vist` | 5,000 | 4,315 | 4,350 |
| | `visual-dialog` | 50,000 | 1,000 | 1,000 |
| | `multi30k` | 90,000 | 6,000 | 12,000 |
| Chinese | `fm-iqa` | 164,735 | 75,206 | 0 |
| | `coco-cn` | 18,341 | 1,000 | 1,000 |
| | `flickr8k-cn` | 6,000 | 1,000 | 1,000 |
| | `chinese-food` | 0 | 0 | 1,100 |
| | `mmchat` | 3,000 | 1,000 | 1,000 |
| Video | `ss` | 2,000 | 2,000 | 2,000 |
| | `ivqa` | 5,994 | 2,000 | 2,000 |
| | `msvd-qa` | 1,161 | 245 | 504 |
| | `activitynet-qa` | 3,200 | 1,800 | 800 |
| | `msrvtt` | 6,513 | 497 | 2,990 |
| | `msrvtt-qa` | 2,000 | 1,000 | 1,000 |
## Dataset Structure
### HuggingFace Login (Optional)
```python
# OR run huggingface-cli login
from huggingface_hub import login
hf_token = "hf_xxx" # TODO: set a valid HuggingFace access token for loading datasets/models
login(token=hf_token)
```
### Data Loading
```python
from datasets import load_dataset
ds_name = "coco" # change the dataset name here
dataset = load_dataset("MMInstruction/M3IT", ds_name)
```
### Data Splits
```python
from datasets import load_dataset
ds_name = "coco" # change the dataset name here
dataset = load_dataset("MMInstruction/M3IT", ds_name)
train_set = dataset["train"]
validation_set = dataset["validation"]
test_set = dataset["test"]
```
### Data Instances
```python
from datasets import load_dataset
from io import BytesIO
from base64 import b64decode
from PIL import Image
ds_name = "coco" # change the dataset name here
dataset = load_dataset("MMInstruction/M3IT", ds_name)
train_set = dataset["train"]
for train_instance in train_set:
instruction = train_instance["instruction"] # str
inputs = train_instance["inputs"] # str
outputs = train_instance["outputs"] # str
image_base64_str_list = train_instance["image_base64_str"] # str (base64)
image_0 = Image.open(BytesIO(b64decode(image_base64_str_list[0])))
```
### Data Fields
```python
import datasets
features = datasets.Features(
{
"instruction": datasets.Value("string"),
"inputs": datasets.Value("string"),
"image_base64_str": [datasets.Value("string")],
"outputs": datasets.Value("string"),
}
)
```
## Dataset Creation
### Curation Rationale
[More Information Needed]
### Source Data
| Task | Dataset [Citation] | Source |
|---------------------------|----------------------------------|------------------------------------------------------------------------------------|
| Image Captioning | `coco` [1] | [Source](https://cocodataset.org/#home) |
| | `textcap` [2] | [Source](https://textvqa.org/textcaps/) |
| | `image-paragraph-captioning` [3] | [Source](https://cs.stanford.edu/people/ranjaykrishna/im2p/index.html) |
| Classification | `coco-goi` [1] | [Source](https://cocodataset.org/#home) |
| | `coco-text` [4] | [Source](https://bgshih.github.io/cocotext/) |
| | `imagenet` [5] | [Source](https://www.image-net.org/) |
| | `coco-itm` [1] | [Source](https://cocodataset.org/#home) |
| | `snli-ve` [6] | [Source](https://github.com/necla-ml/SNLI-VE) |
| | `mocheg` [7] | [Source](https://github.com/VT-NLP/Mocheg) |
| | `iqa` [8] | [Source](https://github.com/icbcbicc/IQA-Dataset) |
| Visual Question Answering | `vqa-v2` [9] | [Source](https://visualqa.org/) |
| | `shapes` [10] | [Source](https://github.com/ronghanghu/n2nmn) |
| | `docvqa` [11] | [Source](https://www.docvqa.org/) |
| | `ocr-vqa` [12] | [Source](https://ocr-vqa.github.io/) |
| | `st-vqa` [13] | [Source](https://rrc.cvc.uab.es/?ch=11) |
| | `text-vqa` [14] | [Source](https://textvqa.org/) |
| | `gqa` [15] | [Source](https://cs.stanford.edu/people/dorarad/gqa/about.html) |
| Knowledgeable Visual QA | `okvqa` [16] | [Source](https://okvqa.allenai.org/) |
| | `a-okvqa` [17] | [Source](https://allenai.org/project/a-okvqa/home) |
| | `science-qa` [18] | [Source](https://scienceqa.github.io/) |
| | `viquae` [19] | [Source](https://github.com/PaulLerner/ViQuAE) |
| Reasoning | `clevr` [20] | [Source](https://cs.stanford.edu/people/jcjohns/clevr/) |
| | `nlvr` [21] | [Source](https://lil.nlp.cornell.edu/nlvr/) |
| | `vcr` [22] | [Source](https://visualcommonsense.com/) |
| | `visual-mrc` [23] | [Source](https://github.com/nttmdlab-nlp/VisualMRC) |
| | `winoground` [24] | [Source](https://huggingface.co/datasets/facebook/winoground) |
| Generation | `vist` [25] | [Source](https://visionandlanguage.net/VIST/) |
| | `visual-dialog` [26] | [Source](https://visualdialog.org/) |
| | `multi30k` [27] | [Source](https://github.com/multi30k/dataset) |
| Chinese | `fm-iqa` [28] | [Source](https://paperswithcode.com/dataset/fm-iqa) |
| | `coco-cn` [29] | [Source](https://github.com/li-xirong/coco-cn) |
| | `flickr8k-cn` [30] | [Source](https://github.com/li-xirong/flickr8kcn) |
| | `chinese-food` [31] | [Source](https://sites.google.com/view/chinesefoodnet) |
| | `mmchat` [32] | [Source](https://github.com/silverriver/MMChat) |
| Video | `ss` [33] | [Source](https://developer.qualcomm.com/software/ai-datasets/something-something) |
| | `ivqa` [34] | [Source](https://antoyang.github.io/just-ask.html) |
| | `msvd-qa` [35] | [Source](https://paperswithcode.com/dataset/msvd) |
| | `activitynet-qa` [36] | [Source](https://github.com/MILVLG/activitynet-qa) |
| | `msrvtt` [35] | [Source](https://paperswithcode.com/dataset/msr-vtt) |
| | `msrvtt-qa` [37] | [Source](https://paperswithcode.com/sota/visual-question-answering-on-msrvtt-qa-1) |
### Annotations
#### Annotation process
To build high-quality multimodal instruction datasets,
we rewrite various datasets into multimodal-to-text dialog format.
The annotation process includes four steps:
- (1) **Stage I: Instruction Writing**: writing instructions for each task;
- (2) **Stage II: Data Format Unification**: structuring images and texts into a unified schema;
- (3) **Stage III: Quality Check**: checking the overall dataset quality;
- (4) **Stage IV: Key Datasets Translation**: building multilingual sets.
#### Who are the annotators?
Eight authors of this work are employed as human annotators,
each of whom is a graduate student familiar with relevant literature.
## Additional Information
### Licensing Information
The content of original dataset follows their original license.
We suggest that for the task with Unknown/Custom license, the user can check the original project or contact the dataset owner for detailed license information.
Our annotated instruction data is licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).
### Citation Information
```bibtex
@article{li2023m3it,
title={M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning},
author={Lei Li and Yuwei Yin and Shicheng Li and Liang Chen and Peiyi Wang and Shuhuai Ren and Mukai Li and Yazheng Yang and Jingjing Xu and Xu Sun and Lingpeng Kong and Qi Liu},
journal={arXiv preprint arXiv:2306.04387},
year={2023}
}
```
### Contributions
M3IT is an open-source, large-scale Multi-modal, Multilingual Instruction Tuning dataset,
designed to enable the development of general-purpose multi-modal agents.
## References
- [1] Microsoft COCO: Common Objects in Context
- [2] TextCaps: a dataset for image captioning with reading comprehension
- [3] A Hierarchical Approach for Generating Descriptive Image Paragraphs
- [4] COCO-Text: Dataset and benchmark for text detection and recognition in natural images
- [5] Imagenet large scale visual recognition challenge
- [6] E-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks
- [7] End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models
- [8] Quantifying visual image quality: A Bayesian view
- [9] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
- [10] Neural Module Networks
- [11] DocVQA: A dataset for vqa on document images
- [12] OCR-VQA: Visual Question Answering by Reading Text in Images
- [13] Scene Text Visual Question Answering
- [14] Towards VQA Models That Can Read
- [15] GQA: A new dataset for real-world visual reasoning and compositional question answering
- [16] OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
- [17] A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge
- [18] Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- [19] ViQuAE: a dataset for knowledge-based visual question answering about named entities
- [20] CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning
- [21] A Corpus of Natural Language for Visual Reasoning
- [22] From recognition to cognition: Visual Commonsense Reasoning
- [23] VisualMRC: Machine reading comprehension on document images
- [24] WinoGround: Probing vision and language models for visio-linguistic compositionality
- [25] Visual Storytelling
- [26] Visual Dialog
- [27] Multi30k: Multilingual english-german image descriptions
- [28] Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question
- [29] COCO-CN for cross-lingual image tagging, captioning, and retrieval
- [30] Adding Chinese Captions to Images
- [31] ChineseFoodNet: A large-scale image dataset for chinese food recognition
- [32] MMChat: Multi-Modal Chat Dataset on Social Media
- [33] The "Something Something" Video Database for Learning and Evaluating Visual Common Sense
- [34] Just Ask: Learning to answer questions from millions of narrated videos
- [35] Video Question Answering via Gradually Refined Attention over Appearance and Motion
- [36] ActivityNet-qa: A dataset for understanding complex web videos via question answering
- [37] MSR-VTT: A large video description dataset for bridging video and language
提供机构：
MMInstruction原始信息汇总
M3IT数据集概述
数据集描述
- 名称: M3IT
- 类别:
- 任务类别:
- 图像到文本
- 图像分类
- 大小类别: 1M<n<10M
- 任务类别:
- 语言: 英语、中文
- 许可: 其他
数据集统计
指令统计
| 任务 | 指令数量 |
|---|---|
| 图像标题生成 | 52 |
| 分类 | 113 |
| 视觉问答 | 95 |
| 知识丰富的视觉问答 | 40 |
| 推理 | 60 |
| 生成 | 40 |
| 总计 | 400 |
任务统计
| 任务 | 描述 | 训练集 | 验证集 | 测试集 |
|---|---|---|---|---|
| 图像标题生成 | 给定图像，为其编写描述 | 679,087 | 41,462 | 27,499 |
| 分类 | 给定图像，将其分类到预定义类别中 | 238,303 | 100,069 | 21,206 |
| 视觉问答 | 给定图像，回答与图像相关的问题 | 177,633 | 46,314 | 10,828 |
| 知识丰富的视觉问答 | 给定图像，回答需要外部知识的问题 | 39,981 | 11,682 | 5,477 |
| 推理 | 给定图像，对图像进行推理 | 99,372 | 11,500 | 10,000 |
| 生成 | 给定图像，根据特定要求进行创作 | 145,000 | 11,315 | 17,350 |
| 中文 | 中文环境下的标题生成、分类、视觉问答和生成任务 | 192,076 | 77,306 | 4,100 |
| 视频 | 视频-语言数据集上的标题生成、分类和视觉问答任务 | 20,868 | 7,542 | 9,294 |
| 多语言 | 80种语言的翻译任务 | 0 | 240,000 | 184,000 |
数据集结构
数据加载
python from datasets import load_dataset
ds_name = "coco" # 在此处更改数据集名称 dataset = load_dataset("MMInstruction/M3IT", ds_name)
数据分割
python from datasets import load_dataset
ds_name = "coco" # 在此处更改数据集名称 dataset = load_dataset("MMInstruction/M3IT", ds_name) train_set = dataset["train"] validation_set = dataset["validation"] test_set = dataset["test"]
数据实例
python from datasets import load_dataset from io import BytesIO from base64 import b64decode from PIL import Image
ds_name = "coco" # 在此处更改数据集名称 dataset = load_dataset("MMInstruction/M3IT", ds_name) train_set = dataset["train"]
for train_instance in train_set: instruction = train_instance["instruction"] # str inputs = train_instance["inputs"] # str outputs = train_instance["outputs"] # str image_base64_str_list = train_instance["image_base64_str"] # str (base64) image_0 = Image.open(BytesIO(b64decode(image_base64_str_list[0])))
数据字段
python import datasets
features = datasets.Features( { "instruction": datasets.Value("string"), "inputs": datasets.Value("string"), "image_base64_str": [datasets.Value("string")], "outputs": datasets.Value("string"), } )
数据集创建
注释过程
- 阶段I: 指令编写
- 阶段II: 数据格式统一
- 阶段III: 质量检查
- 阶段IV: 关键数据集翻译
注释者
八位作者作为人工注释者，每位都是熟悉相关文献的研究生。
附加信息
许可信息
原始数据集内容遵循其原始许可。建议对于未知/自定义许可的任务，用户可以检查原始项目或联系数据集所有者获取详细的许可信息。
我们的注释指令数据根据CC BY 4.0许可。
搜集汇总
数据集介绍
构建方式
M3IT数据集的构建遵循了一套严谨的多阶段流程，旨在将多样化的经典视觉-语言任务统一转化为多模态对话格式。首先，研究团队为图像描述、视觉问答、分类、推理等六大任务类别精心撰写了400条指令模板。随后，通过统一的数据模式将图像与文本结构化，并辅以严格的质量审查，确保数据的一致性与准确性。最后，为拓展多语言能力，团队对关键数据集进行了翻译，生成了包含80种语言的扩展版本。整个标注过程由八位熟悉相关文献的研究生共同完成，保证了标注的专业性。
使用方法
M3IT的使用极为便捷，完全集成于HuggingFace Datasets库中。用户可通过指定数据集名称（如'coco'）直接调用`load_dataset('MMInstruction/M3IT', ds_name)`加载所需子集。数据集预分割为训练集、验证集和测试集，便于标准化评估。每个数据实例包含'instruction'（指令）、'inputs'（输入）、'outputs'（输出）以及以Base64编码的'image_base64_str'图像字段。用户可借助PIL库解码图像，灵活适配各类多模态模型的训练与微调流程。
背景与挑战
背景概述
随着多模态大语言模型的蓬勃发展，如何构建能够统一处理图像、视频与文本等多种模态信息并遵循人类指令的通用智能体成为研究焦点。在此背景下，由北京大学、西湖大学等机构的研究人员于2023年提出的M3IT（Multi-Modal Multilingual Instruction Tuning）数据集应运而生。该数据集的核心研究问题在于如何系统性地整合视觉语言任务中的指令微调数据，以促进模型对多样化任务的理解与泛化能力。M3IT汇聚了图像描述、视觉问答、分类、推理、生成等9大类任务，涵盖超过40个经典子数据集，总指令数达400条，训练样本近240万，并提供了中英双语及80种多语言扩展版本。其发布为多模态指令微调领域提供了标准化、大规模的数据基础，显著推动了通用多模态对话系统的发展。
当前挑战
M3IT所解决的领域挑战在于多模态模型在指令遵循与任务泛化上的不足。传统视觉语言模型往往针对单一任务训练，缺乏对多样化、开放式指令的响应能力，而M3IT通过统一指令格式和任务覆盖，旨在突破这一瓶颈，使模型能够同时处理图像分类、视觉推理、知识问答等异构任务。在构建过程中，团队面临的核心挑战包括：其一，来自不同来源的原始数据集格式、标注规范与任务定义高度异构，需设计统一的指令化重构流程，将图像、文本与输出映射为多模态对话结构；其二，多语言扩展中需确保翻译质量与任务语义的保真度，避免因语言转换导致指令歧义或信息丢失；其三，大规模数据质量校验的自动化与人工审核平衡，需在400条指令和近240万样本中过滤噪声、纠正错标，并维护任务间的覆盖均衡性。
常用场景
经典使用场景
M3IT作为大规模多模态多语言指令微调数据集，最经典的使用场景在于统一化视觉与语言任务的指令格式，推动通用多模态智能体的训练。该数据集整合了图像描述、视觉问答、分类、推理与生成等核心任务，提供超过240万条训练样本，覆盖英文与中文，并扩展至80种语言。研究者可借助其统一的结构化接口，高效构建能够理解图像、文本并执行多样化指令的模型，从而在少样本与零样本场景下实现跨任务迁移。这一设计显著降低了多模态指令微调的数据门槛，成为评估与提升多模态大模型指令跟随能力的标准基准。
解决学术问题
M3IT系统性地解决了多模态指令微调领域长期存在的两大核心瓶颈：任务覆盖碎片化与语言多样性缺失。此前，视觉语言数据集多聚焦于单一任务（如仅包含VQA或仅包含描述），缺乏统一格式，导致模型难以泛化至多种指令模式。M3IT通过标准化400余条指令，将图像描述、视觉问答、知识驱动推理、视觉分类与条件生成等任务整合为一致的“图像-指令-输出”结构，使模型得以学习跨任务共享的视觉语言关联。此外，该数据集引入80种语言的翻译版本，填补了非英语多模态指令数据的空白，为多语言多模态模型的公平性评估与全球化部署提供了关键资源，推动学术界从任务特定模型向通用智能体的范式转变。
实际应用
在实际应用中，M3IT驱动的多模态指令微调模型可直接赋能智能客服、无障碍辅助与内容创作等场景。例如，在智能客服领域，模型能够依据用户上传的图片与自然语言指令，自动生成产品描述、解答视觉相关问题或执行分类操作；在无障碍辅助中，模型可为视障人士提供图像内容的口头描述或上下文推理；在内容创作平台，模型可基于图像生成故事、多语言字幕或对话回应。由于M3IT覆盖了中文及80种语言的数据，这些应用可无缝适配全球市场，显著降低多语言多模态系统的开发成本，提升人机交互的直观性与效率。
数据集最近研究
最新研究方向
M3IT作为大规模多模态多语言指令微调数据集，正推动视觉语言模型向通用智能体方向发展。当前前沿研究聚焦于利用其400余项指令和超百万样本，探索模型在图像描述、视觉问答、知识推理等任务上的跨模态对齐能力。该数据集涵盖中英文及80种语言翻译版本，为多语言视觉语言模型的训练与评估提供了坚实基础。在热点事件上，M3IT与大语言模型（如GPT-4V）的协同研究备受关注，旨在通过指令微调增强模型对复杂视觉场景的理解和生成能力。其意义在于打破了传统任务间的壁垒，促进了视觉语言基础模型的泛化性和零样本学习能力，为构建更智能、更包容的多模态AI系统奠定了关键数据基础。
以上内容由遇见数据集搜集并总结生成
