Ryoo72/MMBench-EN-Dev-V10

---
language:
- en
license: apache-2.0
task_categories:
- visual-question-answering
- multiple-choice
tags:
- mmbench
- multimodal
- vlm-eval
size_categories:
- 1K<n<10K
configs:
- config_name: all
data_files:
- split: dev
path: all/dev-*
- config_name: coarse_perception
data_files:
- split: dev
path: coarse_perception/dev-*
- config_name: finegrained_perception_single_instance
data_files:
- split: dev
path: finegrained_perception_single_instance/dev-*
- config_name: finegrained_perception_cross_instance
data_files:
- split: dev
path: finegrained_perception_cross_instance/dev-*
- config_name: attribute_reasoning
data_files:
- split: dev
path: attribute_reasoning/dev-*
- config_name: relation_reasoning
data_files:
- split: dev
path: relation_reasoning/dev-*
- config_name: logic_reasoning
data_files:
- split: dev
path: logic_reasoning/dev-*
---

# MMBench EN Dev V1.0 — split by L2 category

The **MMBench-EN-Dev-V1.0** dev split, pre-split into 6 subsets by the original `l2-category` field for convenient browsing in the dataset viewer.

- **Source**: official OpenCompass TSV [`MMBench_DEV_EN.tsv`](http://opencompass.openxlab.space/utils/benchmarks/MMBench/MMBench_DEV_EN.tsv)
- **MD5 (verified)**: `b6caf1133a01c6bb705cf753bb527ed8` — matches the value registered in [VLMEvalKit](https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/image_mcq.py).
- **Rows**: 4,329
- **Format note**: the source TSV uses VLMEvalKit's compressed CircularEval format where many rows reference another row's image by `index` instead of storing base64. References are **resolved** here, so every row carries its own decoded `image`.

## Subsets

| Config | Source `l2-category` | # samples |
|---|---|---:|
| `all` | (everything) | 4,329 |
| `coarse_perception` | `coarse_perception` | 1,109 |
| `finegrained_perception_single_instance` | `finegrained_perception (instance-level)` | 1,131 |
| `finegrained_perception_cross_instance` | `finegrained_perception (cross-instance)` | 533 |
| `attribute_reasoning` | `attribute_reasoning` | 699 |
| `relation_reasoning` | `relation_reasoning` | 445 |
| `logic_reasoning` | `logic_reasoning` | 412 |

Each subset has a single split: `dev`. The L3 ability ladder (20 fine-grained classes) is preserved in the `category` column.

## Columns

`index`, `question`, `hint`, `A`, `B`, `C`, `D`, `answer`, `category` (L3, 20 classes), `image` (PIL), `l2-category` (L2, 6 classes), `split`, `source`, `comment`.

## Usage

```python
from datasets import load_dataset
ds = load_dataset("Ryoo72/MMBench-EN-Dev-V10", "all", split="dev")
ds_logic = load_dataset("Ryoo72/MMBench-EN-Dev-V10", "logic_reasoning", split="dev")
```

## Related

- **V1.1**: [Ryoo72/MMBench-EN-Dev-V11](https://huggingface.co/datasets/Ryoo72/MMBench-EN-Dev-V11)
- Paper: [MMBench: Is Your Multi-modal Model an All-around Player?](https://arxiv.org/abs/2307.06281)
- Project: <https://opencompass.org.cn/mmbench>
- Eval kit: <https://github.com/open-compass/VLMEvalKit>