changdae/vittle-llavabench-coco-joint-perturbed

---
license: mit
task_categories:
- visual-question-answering
tags:
- robustness
- LLaVA-Bench
- COCO
- perturbation
- vittle
- joint-perturbation
pretty_name: "Vittle - Jointly Perturbed LLaVA-Bench-COCO"
size_categories:
- n<1K
---

# Vittle - Jointly Perturbed LLaVA-Bench-COCO

This dataset provides **jointly perturbed** (visual + textual) variants of the [LLaVA-Bench (COCO)](https://arxiv.org/abs/2304.08485) open-ended VQA benchmark.
It is released as part of the [Vittle (Visual Instruction Bottleneck Tuning)](https://arxiv.org/abs/2505.13946) project (NeurIPS 2025).

## Overview

- **Questions**: 90 base questions x 3 textual perturbation variants (translations)
- **Images**: 30 unique COCO val2014 images x 3 visual perturbation variants
- **Joint combinations**: 3 visual x 3 textual = 9 joint perturbation settings

## Perturbation Combinations

### Visual Perturbations (severity level 3)
Generated following [MM-Robustness](https://github.com/Jielin-Qiu/MM_Robustness):
- Gaussian Noise (`COCO_IP_gaussian_noise_3`)
- Zoom Blur (`COCO_IP_zoom_blur_3`)
- Frost (`COCO_IP_frost_3`)

### Textual Perturbations (translation via GPT-4o)
- Hindi (`qa90_questions_Hindi.jsonl`)
- Greek (`qa90_questions_Greek.jsonl`)
- Arabic (`qa90_questions_Arabic.jsonl`)

### All 9 Joint Settings
| Visual \ Textual | Hindi | Greek | Arabic |
|---|---|---|---|
| Gaussian Noise | Y | Y | Y |
| Zoom Blur | Y | Y | Y |
| Frost | Y | Y | Y |

## File Structure

```
.
├── README.md
├── qa90_questions.jsonl # 90 original (clean) questions
├── questions_perturbed/
│ ├── qa90_questions_Hindi.jsonl
│ ├── qa90_questions_Greek.jsonl
│ └── qa90_questions_Arabic.jsonl
└── images/
├── COCO_IP_gaussian_noise_3/ # 30 images
├── COCO_IP_zoom_blur_3/
└── COCO_IP_frost_3/
```

## Usage

To evaluate a model on a joint perturbation setting, pair a perturbed question file with a perturbed image folder. For example, for (Frost + Hindi):
- Questions: `questions_perturbed/qa90_questions_Hindi.jsonl`
- Images: `images/COCO_IP_frost_3/`

## Citation

```bibtex
@inproceedings{
oh2025visual,
title={Visual Instruction Bottleneck Tuning},
author={Changdae Oh and Jiatong Li and Shawn Im and Sharon Li},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=yzHiEmLSk8}
}
```

## License

MIT