Ryoo72/MMBench-EN-Dev-V11

---
language:
- en
license: apache-2.0
task_categories:
- visual-question-answering
- multiple-choice
tags:
- mmbench
- multimodal
- vlm-eval
size_categories:
- 1K<n<10K
configs:
- config_name: all
data_files:
- split: dev
path: all/dev-*
- config_name: coarse_perception
data_files:
- split: dev
path: coarse_perception/dev-*
- config_name: finegrained_perception_single_instance
data_files:
- split: dev
path: finegrained_perception_single_instance/dev-*
- config_name: finegrained_perception_cross_instance
data_files:
- split: dev
path: finegrained_perception_cross_instance/dev-*
- config_name: attribute_reasoning
data_files:
- split: dev
path: attribute_reasoning/dev-*
- config_name: relation_reasoning
data_files:
- split: dev
path: relation_reasoning/dev-*
- config_name: logic_reasoning
data_files:
- split: dev
path: logic_reasoning/dev-*
---

# MMBench EN Dev V1.1 — split by L2 category

The **MMBench-EN-Dev-V1.1** dev split, pre-split into 6 subsets by the original `l2-category` field for convenient browsing in the dataset viewer.

- **Source**: official OpenCompass TSV [`MMBench_DEV_EN_V11.tsv`](http://opencompass.openxlab.space/utils/benchmarks/MMBench/MMBench_DEV_EN_V11.tsv)
- **MD5 (verified)**: `30c05be8f2f347a50be25aa067248184` — matches the value registered in [VLMEvalKit](https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/image_mcq.py).
- **Rows**: 4,876
- **Format note**: the source TSV uses VLMEvalKit's compressed CircularEval format where many rows reference another row's image by `index` instead of storing base64. References are **resolved** here, so every row carries its own decoded `image`.

## Subsets

| Config | Source `l2-category` | # samples |
|---|---|---:|
| `all` | (everything) | 4,876 |
| `coarse_perception` | `coarse_perception` | 1,381 |
| `finegrained_perception_single_instance` | `finegrained_perception (instance-level)` | 1,128 |
| `finegrained_perception_cross_instance` | `finegrained_perception (cross-instance)` | 667 |
| `attribute_reasoning` | `attribute_reasoning` | 603 |
| `relation_reasoning` | `relation_reasoning` | 637 |
| `logic_reasoning` | `logic_reasoning` | 460 |

Each subset has a single split: `dev`. The L3 ability ladder (20 fine-grained classes) is preserved in the `category` column.

## Columns

`index`, `question`, `hint`, `A`, `B`, `C`, `D`, `answer`, `category` (L3, 20 classes), `image` (PIL), `l2-category` (L2, 6 classes), `split`.

## Usage

```python
from datasets import load_dataset
ds = load_dataset("Ryoo72/MMBench-EN-Dev-V11", "all", split="dev")
ds_logic = load_dataset("Ryoo72/MMBench-EN-Dev-V11", "logic_reasoning", split="dev")
```

## Related

- **V1.0**: [Ryoo72/MMBench-EN-Dev-V10](https://huggingface.co/datasets/Ryoo72/MMBench-EN-Dev-V10)
- Paper: [MMBench: Is Your Multi-modal Model an All-around Player?](https://arxiv.org/abs/2307.06281)
- Project: <https://opencompass.org.cn/mmbench>
- Eval kit: <https://github.com/open-compass/VLMEvalKit>