CODIS/CODIS

---
license: apache-2.0
task_categories:
- visual-question-answering
language:
- en
pretty_name: CODIS
size_categories:
- n<1K
---
# CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models

[**🌐 Homepage**](https://thunlp-mt.github.io/CODIS) | [**📖 arXiv**](https://arxiv.org/abs/2402.13607) | [**Github**](https://github.com/THUNLP-MT/CODIS)

Dataset for paper [CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models](https://arxiv.org/abs/2402.13607).

## Introduction

In certain situations, images need to be interpreted within a broader context. We introduce a new benchmark, named as **CODIS** (**CO**ntext-**D**ependent **I**mage di**S**ambiguation), designed to assess the ability of models to use context provided in free-form text to enhance visual comprehension.

- Each image in CODIS contains inherent ambiguity that can only be resolved with additional context.
- The questions are deliberately designed to highlight these ambiguities, requiring external context for accurate interpretation.
- For every image-question pair, we provide two contexts in a free-form text format.

## Leaderboard

We report results of human and MLLMs. Models score only if their answers to a pair of queries are both correct. The results are based on human evaluation.

| Model | Loc & Ori | Temporal | Cultural | Attributes | Relationships | Average |
|----------------|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|
Human | 85.2 | 90.9 | 72.8 | 87.2 | 89.6 | 86.2 |
GPT4-V | 33.3 | 28.4 | 25.5 | 26.7 | 51.9 | 32.3 |
Gemini | 21.4 | 29.5 | 21.3 | 24.0 | 34.6 | 26.1 |
LLaVA-1.5-13B | 6.0 | 4.2 | 10.6 | 14.7 | 13.5 | 9.1 |
BLIP-2-11B | 6.0 | 8.4 | 4.3 | 6.7 | 11.5 | 7.4 |
InstructBLIP-13B | 6.0 | 2.1 | 4.3 | 4.0 | 7.7 | 4.5 |
mPLUG-Owl-2-7B | 13.1 | 9.5 | 6.4 | 12.0 | 19.2 | 11.9 |
MiniGPT4-7B | 10.7 | 3.2 | 0.0 | 12.0 | 13.5 | 7.9 |
LLaVA-1.5-7B | 11.9 | 5.3 | 4.3 | 9.3 | 7.7 | 7.9 |
InstructBLIP-7B | 1.2 | 7.4 | 0.0 | 4.0 | 11.5 | 4.8 |
Otter-7B | 2.4 | 5.3 | 4.3 | 0.0 | 5.8 | 3.4 |
LLaVA-7B | 2.4 | 6.3 | 0.0 | 1.3 | 5.8 | 3.4 |
Qwen-VL-Chat | 3.6 | 3.2 | 0.0 | 1.3 | 9.6 | 3.4 |
OpenFlamingo-7B | 2.4 | 2.1 | 0.0 | 5.3 | 5.8 | 3.1 |
BLIP-2-6.7B | 0.0 | 1.1 | 2.1 | 2.7 | 7.7 | 2.3 |

## Citation

```bibtex
@article{luo2024codis,
title={CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models},
author={Fuwen Luo and Chi Chen and Zihao Wan and Zhaolu Kang and Qidong Yan and Yingjie Li and Xiaolong Wang and Siyu Wang and Ziyue Wang and Xiaoyue Mi and Peng Li and Ning Ma and Maosong Sun and Yang Liu},
journal={arXiv preprint arXiv:2402.13607},
year={2024}
}
```

---
许可证：apache-2.0
任务类别：
- 视觉问答（visual-question-answering）
语言：
- 英语
美观名称：CODIS
样本量级：
- n<1K
---
# CODIS：面向多模态大语言模型的上下文依赖视觉理解基准测试

[🌐 项目主页](https://thunlp-mt.github.io/CODIS) | [📖 arXiv预印本](https://arxiv.org/abs/2402.13607) | [📦 Github仓库](https://github.com/THUNLP-MT/CODIS)

本数据集配套论文《CODIS：面向多模态大语言模型的上下文依赖视觉理解基准测试》，链接：[arXiv预印本](https://arxiv.org/abs/2402.13607)。

## 简介

在部分实际场景中，图像的解读需要结合更宽泛的上下文信息。我们推出了一款全新基准测试集**CODIS**（全称为**CO**ntext-**D**ependent **I**mage di**S**ambiguation，即上下文依赖图像消歧基准），旨在评估多模态大语言模型（Multimodal Large Language Model，MLLM）利用自由格式文本提供的上下文信息以提升视觉理解能力的性能。

- CODIS中的每幅图像均存在固有歧义，仅能通过补充上下文信息加以消解
- 所有问题均经过精心设计以凸显这类歧义，需要借助外部上下文才能实现准确解读
- 针对每一组图像-问题对，我们均提供两份自由格式文本形式的上下文信息

## 排行榜

我们在此报告人类评估者与多模态大语言模型的测试结果。模型仅当其针对一组查询的两项回答均正确时方可获得得分，本次实验结果均基于人工评估得出。

| 模型名称 | 位置与方向 | 时序信息 | 文化背景 | 属性特征 | 语义关系 | 平均得分 |
|------------------------|:---------:|:-------:|:-------:|:-------:|:-------:|:-------:|
| 人类评估者 | 85.2 | 90.9 | 72.8 | 87.2 | 89.6 | 86.2 |
| GPT4-V | 33.3 | 28.4 | 25.5 | 26.7 | 51.9 | 32.3 |
| Gemini | 21.4 | 29.5 | 21.3 | 24.0 | 34.6 | 26.1 |
| LLaVA-1.5-13B | 6.0 | 4.2 | 10.6 | 14.7 | 13.5 | 9.1 |
| BLIP-2-11B | 6.0 | 8.4 | 4.3 | 6.7 | 11.5 | 7.4 |
| InstructBLIP-13B | 6.0 | 2.1 | 4.3 | 4.0 | 7.7 | 4.5 |
| mPLUG-Owl-2-7B | 13.1 | 9.5 | 6.4 | 12.0 | 19.2 | 11.9 |
| MiniGPT4-7B | 10.7 | 3.2 | 0.0 | 12.0 | 13.5 | 7.9 |
| LLaVA-1.5-7B | 11.9 | 5.3 | 4.3 | 9.3 | 7.7 | 7.9 |
| InstructBLIP-7B | 1.2 | 7.4 | 0.0 | 4.0 | 11.5 | 4.8 |
| Otter-7B | 2.4 | 5.3 | 4.3 | 0.0 | 5.8 | 3.4 |
| LLaVA-7B | 2.4 | 6.3 | 0.0 | 1.3 | 5.8 | 3.4 |
| Qwen-VL-Chat | 3.6 | 3.2 | 0.0 | 1.3 | 9.6 | 3.4 |
| OpenFlamingo-7B | 2.4 | 2.1 | 0.0 | 5.3 | 5.8 | 3.1 |
| BLIP-2-6.7B | 0.0 | 1.1 | 2.1 | 2.7 | 7.7 | 2.3 |

## 引用格式

bibtex
@article{luo2024codis,
title={CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models},
author={Fuwen Luo and Chi Chen and Zihao Wan and Zhaolu Kang and Qidong Yan and Yingjie Li and Xiaolong Wang and Siyu Wang and Ziyue Wang and Xiaoyue Mi and Peng Li and Ning Ma and Maosong Sun and Yang Liu},
journal={arXiv preprint arXiv:2402.13607},
year={2024}
}

原始信息汇总

数据集概述

本数据集详情页面提供了以下资源链接：

• Homepage: 数据集的主页链接。
• arXiv: 数据集相关的arXiv论文链接。
• Github: 数据集的Github仓库链接。