changdae/vittle-llavabench-coco-visual-perturbed

--- license: mit task_categories: - visual-question-answering tags: - robustness - LLaVA-Bench - COCO - perturbation - vittle pretty_name: "Vittle - Visually Perturbed LLaVA-Bench-COCO" size_categories: - n<1K --- # Vittle - Visually Perturbed LLaVA-Bench-COCO This dataset provides **visually perturbed** variants of the [LLaVA-Bench (COCO)](https://arxiv.org/abs/2304.08485) open-ended VQA benchmark. It is released as part of the [Vittle (Visual Instruction Bottleneck Tuning)](https://arxiv.org/abs/2505.13946) project (NeurIPS 2025). ## Overview - **Questions**: 90 open-ended questions (conversation, detail, complex) — clean text, perturbed images - **Images**: 30 unique COCO val2014 images, each with 9 visual perturbation variants (severity level 3) - **Total image files**: 270 (30 images x 9 perturbations) ## Visual Perturbations All perturbations are at severity level 3, generated following [MM-Robustness](https://github.com/Jielin-Qiu/MM_Robustness): | Perturbation | Folder | |---|---| | Gaussian Noise | `images/COCO_IP_gaussian_noise_3/` | | Shot Noise | `images/COCO_IP_shot_noise_3/` | | Speckle Noise | `images/COCO_IP_speckle_noise_3/` | | Fog | `images/COCO_IP_fog_3/` | | Contrast | `images/COCO_IP_contrast_3/` | | Brightness | `images/COCO_IP_brightness_3/` | | Defocus Blur | `images/COCO_IP_defocus_blur_3/` | | Zoom Blur | `images/COCO_IP_zoom_blur_3/` | | Frost | `images/COCO_IP_frost_3/` | ## File Structure ``` . ├── README.md ├── qa90_questions.jsonl # 90 questions (clean text) └── images/ ├── COCO_IP_gaussian_noise_3/ # 30 images ├── COCO_IP_shot_noise_3/ ├── COCO_IP_speckle_noise_3/ ├── COCO_IP_fog_3/ ├── COCO_IP_contrast_3/ ├── COCO_IP_brightness_3/ ├── COCO_IP_defocus_blur_3/ ├── COCO_IP_zoom_blur_3/ └── COCO_IP_frost_3/ ``` ## Question Format (JSONL) ```json {"question_id": 1, "image": "COCO_val2014_000000367571.jpg", "text": "What are the colors of the bus in the image?", "category": "conv"} ``` ## Citation ```bibtex @inproceedings{oh2025vittle, title={Visual Instruction Bottleneck Tuning}, author={Oh, Changdae and Li, Jiatong and Im, Shawn and Li, Yixuan}, booktitle={Advances in Neural Information Processing Systems}, year={2025} } ``` ## License MITlicense: MIT许可证 task_categories: - 视觉问答（Visual Question Answering, VQA） tags: - 鲁棒性（robustness） - LLaVA-Bench - COCO - 扰动（perturbation） - vittle pretty_name: "Vittle——带视觉扰动的LLaVA-Bench-COCO数据集" size_categories: - 样本量小于1000 --- # Vittle——带视觉扰动的LLaVA-Bench-COCO数据集 本数据集提供了**视觉扰动（visually perturbed）**变体的[LLaVA-Bench（COCO）](https://arxiv.org/abs/2304.08485)开放式视觉问答基准测试集。本数据集作为[Vittle（视觉指令瓶颈调优，Visual Instruction Bottleneck Tuning）](https://arxiv.org/abs/2505.13946)项目（NeurIPS 2025）的一部分发布。 ## 概览 - **问题集**：共90道开放式问题（涵盖对话、细节、复杂推理三类）——采用干净文本提问，搭配经过视觉扰动的图像 - **图像集**：包含30张独特的COCO val2014图像，每张图像对应9种视觉扰动变体（扰动强度等级为3） - **总图像文件数**：270张（30张原始图像 × 9种扰动类型） ## 视觉扰动说明 所有扰动均采用强度等级3，生成方法遵循[MM-Robustness](https://github.com/Jielin-Qiu/MM_Robustness)工具链： | 扰动类型 | 文件夹路径 | |---|---| | 高斯噪声（Gaussian Noise） | `images/COCO_IP_gaussian_noise_3/` | | 散粒噪声（Shot Noise） | `images/COCO_IP_shot_noise_3/` | | 斑点噪声（Speckle Noise） | `images/COCO_IP_speckle_noise_3/` | | 雾化效果（Fog） | `images/COCO_IP_fog_3/` | | 对比度调整（Contrast） | `images/COCO_IP_contrast_3/` | | 亮度调整（Brightness） | `images/COCO_IP_brightness_3/` | | 散焦模糊（Defocus Blur） | `images/COCO_IP_defocus_blur_3/` | | 变焦模糊（Zoom Blur） | `images/COCO_IP_zoom_blur_3/` | | 霜冻效果（Frost） | `images/COCO_IP_frost_3/` | ## 文件结构 . ├── README.md ├── qa90_questions.jsonl # 90道问题（干净文本格式） └── images/ ├── COCO_IP_gaussian_noise_3/ # 30张扰动图像 ├── COCO_IP_shot_noise_3/ ├── COCO_IP_speckle_noise_3/ ├── COCO_IP_fog_3/ ├── COCO_IP_contrast_3/ ├── COCO_IP_brightness_3/ ├── COCO_IP_defocus_blur_3/ ├── COCO_IP_zoom_blur_3/ └── COCO_IP_frost_3/ ## 问题格式（JSONL） json {"question_id": 1, "image": "COCO_val2014_000000367571.jpg", "text": "What are the colors of the bus in the image?", "category": "conv"} ## 引用格式 bibtex @inproceedings{oh2025vittle, title={Visual Instruction Bottleneck Tuning}, author={Oh, Changdae and Li, Jiatong and Im, Shawn and Li, Yixuan}, booktitle={Advances in Neural Information Processing Systems}, year={2025} } ## 许可证 MIT许可证