UniMM-Chat

# Dataset Card for UniMM-Chat

## Dataset Summary

UniMM-Chat dataset is an **open-source, knowledge-intensive, and multi-round multimodal dialogue data** powered by GPT-3.5, which consists of **1.1M diverse instructions**.

UniMM-Chat leverages **complementary annotations from different VL datasets** and employs GPT-3.5 to generate multi-turn dialogues corresponding to each image, resulting in **117,238 dialogues**, with an average of **9.89 turns per dialogue**.

<p align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/6566e0c493e30c8a60048eb3/HQlP6gRsIq9E2czvmunca.png" alt="fig1" width="60%"/>
</p>

**A diverse set of instructions**:

<p align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/6566e0c493e30c8a60048eb3/8gmR9FWnCjDIs8IQ7ZxpU.png" alt="fig1" width="30%"/>
</p>

**Resulting superior performance in image understanding and reasoning**:

<p align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/6566e0c493e30c8a60048eb3/YZceD395gErU7FiVVBljE.png" alt="fig1" width="40%"/>
</p>

## Related Sources

- Paper: https://arxiv.org/abs/2310.00653
- Models Trained on UniMM-Chat: 🥞[Muffin](https://github.com/thunlp/muffin), 🏆[RLHF-V](https://rlhf-v.github.io)

## Usage

```python
from datasets import load_dataset
data = load_dataset("Yirany/UniMM-Chat")
```

## Citation

```
@article{yu2023reformulating,
title={Reformulating vision-language foundation models and datasets towards universal multimodal assistants},
author={Yu, Tianyu and Hu, Jinyi and Yao, Yuan and Zhang, Haoye and Zhao, Yue and Wang, Chongyi and Wang, Shan and Pan, Yinxv and Xue, Jiao and Li, Dahai and others},
journal={arXiv preprint arXiv:2310.00653},
year={2023}
}
```

# UniMM-Chat 数据集卡片

## 数据集概述
UniMM-Chat数据集是一款由GPT-3.5驱动的**开源、知识密集型多轮多模态对话数据集**，共包含**110万条多样化指令**。

该数据集利用了不同视觉语言（Vision-Language, VL）数据集的互补标注，并借助GPT-3.5为每张图像生成对应多轮对话，最终得到**117238条对话**，单条对话平均轮次为**9.89轮**。

<p align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/6566e0c493e30c8a60048eb3/HQlP6gRsIq9E2czvmunca.png" alt="fig1" width="60%"/>
</p>

**多样化指令集**：

<p align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/6566e0c493e30c8a60048eb3/8gmR9FWnCjDIs8IQ7ZxpU.png" alt="fig1" width="30%"/>
</p>

**在图像理解与推理任务中表现优异**：

<p align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/6566e0c493e30c8a60048eb3/YZceD395gErU7FiVVBljE.png" alt="fig1" width="40%"/>
</p>

## 相关资源
- 论文：https://arxiv.org/abs/2310.00653
- 基于UniMM-Chat训练的模型：🥞[Muffin](https://github.com/thunlp/muffin)、🏆[RLHF-V](https://rlhf-v.github.io)

## 使用方法
python
from datasets import load_dataset
data = load_dataset("Yirany/UniMM-Chat")

## 引用格式

@article{yu2023reformulating,
title={Reformulating vision-language foundation models and datasets towards universal multimodal assistants},
author={Yu, Tianyu and Hu, Jinyi and Yao, Yuan and Zhang, Haoye and Zhao, Yue and Wang, Chongyi and Wang, Shan and Pan, Yinxv and Xue, Jiao and Li, Dahai and others},
journal={arXiv preprint arXiv:2310.00653},
year={2023}
}