LLaVA-NeXT-Interleave-Bench

# LLaVA-Interleave Bench Dataset Card

## Dataset details

**Dataset type:**
LLaVA-Interleave Bench is a comprehensive set of multi-image datasets that are collected from public datasets or generated by the GPT-4V API.
It is constructed for evaluating the interleaved multi-image reaoning capbilities of LMMs.

**Dataset date:**
LLaVA-Interleave Bench was collected in April 2024, and released in June 2024.

**Paper or resources for more information:**
Blog: https://llava-vl.github.io/blog/2024-06-16-llava-next-interleave/

**Evaluate LLaVA-NeXT Interleave Model**

### Preparation

Please download the evaluation data first.

Unzip eval_images.zip and there are Split1 and Split2 in it.
Organize the downloaded data into the following structure:
```
interleave_data
├── Split1
│ ├── ...
│ └── ...
|
├── Split2
| ├── ...
│ └── ...
├── multi_image_in_domain.json
├── multi_image_out_domain.json
└── multi_view_in_domain.json
```
### Inference and Evaluation
Example:
Please first edit /path/to/ckpt to the path of checkpoint, /path/to/images to the path of "interleave_data" in scripts/interleave/eval_all.sh and then run
```bash
bash scripts/interleave/eval_all.sh
```
Note that, the MMMU-mv data is currently not included in the released data. We will release it soon. To construct MMMU-mv, we sample the 789 questions with multiple input images from the [official test set](https://huggingface.co/datasets/MMMU/MMMU) of MMMU.

**License:**
Creative Commons Attribution 4.0 International; and it should abide by the policy of OpenAI: https://openai.com/policies/terms-of-use
**Where to send questions or comments about the model:**
[e-mail]
## Intended use
**Primary intended uses:**
The primary use of LLaVA-Next Interleave is research on large multimodal models and chatbots.
**Primary intended users:**
The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.

# LLaVA-Interleave Bench 数据集卡片

## 数据集详情

**数据集类型：**
LLaVA-Interleave Bench 是一套综合性多图像数据集，其数据源自公开数据集或通过 GPT-4V API 生成，旨在评估多模态大模型（Large Multimodal Model, LMM）的交错多图像推理能力。

**数据集采集与发布时间：**
本数据集于2024年4月完成采集，2024年6月正式发布。

**更多信息参考的论文或资源：**
博客：https://llava-vl.github.io/blog/2024-06-16-llava-next-interleave/

**评估 LLaVA-NeXT Interleave 模型**

### 准备工作
请先下载评估数据。
解压 eval_images.zip 后，内部将包含 Split1 与 Split2 两个子目录。请将下载的数据按如下结构组织：

interleave_data
├── Split1
│ ├── ...
│ └── ...
|
├── Split2
| ├── ...
│ └── ...
├── multi_image_in_domain.json
├── multi_image_out_domain.json
└── multi_view_in_domain.json

### 推理与评估
示例：
请先在 scripts/interleave/eval_all.sh 脚本中，将 /path/to/ckpt 修改为模型检查点路径，将 /path/to/images 修改为上述 interleave_data 目录的路径，随后运行如下命令：
bash
bash scripts/interleave/eval_all.sh

注意：当前发布的数据未包含 MMMU-mv 数据集，我们将尽快推出该数据集的发布。MMMU-mv 的构建方式为：从 MMMU 的[官方测试集](https://huggingface.co/datasets/MMMU/MMMU)中采样789道包含多幅输入图像的问题。

**许可证：**
知识共享署名4.0国际许可协议（Creative Commons Attribution 4.0 International）；同时需遵守 OpenAI 的相关政策：https://openai.com/policies/terms-of-use
**问题与意见反馈渠道：**
[e-mail]

## 预期用途
**主要预期用途：**
LLaVA-Next Interleave 的核心用途为多模态大模型与聊天机器人相关的研究工作。
**主要目标用户群体：**
本模型的主要目标用户为计算机视觉、自然语言处理、机器学习与人工智能领域的研究人员与爱好者。