changdae/vittle-llavabench-coco-joint-perturbed

--- license: mit task_categories: - visual-question-answering tags: - robustness - LLaVA-Bench - COCO - perturbation - vittle - joint-perturbation pretty_name: "Vittle - Jointly Perturbed LLaVA-Bench-COCO" size_categories: - n<1K --- # Vittle - Jointly Perturbed LLaVA-Bench-COCO This dataset provides **jointly perturbed** (visual + textual) variants of the [LLaVA-Bench (COCO)](https://arxiv.org/abs/2304.08485) open-ended VQA benchmark. It is released as part of the [Vittle (Visual Instruction Bottleneck Tuning)](https://arxiv.org/abs/2505.13946) project (NeurIPS 2025). ## Overview - **Questions**: 90 base questions x 3 textual perturbation variants (translations) - **Images**: 30 unique COCO val2014 images x 3 visual perturbation variants - **Joint combinations**: 3 visual x 3 textual = 9 joint perturbation settings ## Perturbation Combinations ### Visual Perturbations (severity level 3) Generated following [MM-Robustness](https://github.com/Jielin-Qiu/MM_Robustness): - Gaussian Noise (`COCO_IP_gaussian_noise_3`) - Zoom Blur (`COCO_IP_zoom_blur_3`) - Frost (`COCO_IP_frost_3`) ### Textual Perturbations (translation via GPT-4o) - Hindi (`qa90_questions_Hindi.jsonl`) - Greek (`qa90_questions_Greek.jsonl`) - Arabic (`qa90_questions_Arabic.jsonl`) ### All 9 Joint Settings | Visual \ Textual | Hindi | Greek | Arabic | |---|---|---|---| | Gaussian Noise | Y | Y | Y | | Zoom Blur | Y | Y | Y | | Frost | Y | Y | Y | ## File Structure ``` . ├── README.md ├── qa90_questions.jsonl # 90 original (clean) questions ├── questions_perturbed/ │ ├── qa90_questions_Hindi.jsonl │ ├── qa90_questions_Greek.jsonl │ └── qa90_questions_Arabic.jsonl └── images/ ├── COCO_IP_gaussian_noise_3/ # 30 images ├── COCO_IP_zoom_blur_3/ └── COCO_IP_frost_3/ ``` ## Usage To evaluate a model on a joint perturbation setting, pair a perturbed question file with a perturbed image folder. For example, for (Frost + Hindi): - Questions: `questions_perturbed/qa90_questions_Hindi.jsonl` - Images: `images/COCO_IP_frost_3/` ## Citation ```bibtex @inproceedings{ oh2025visual, title={Visual Instruction Bottleneck Tuning}, author={Changdae Oh and Jiatong Li and Shawn Im and Sharon Li}, booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems}, year={2025}, url={https://openreview.net/forum?id=yzHiEmLSk8} } ``` ## License MIT许可证：MIT 任务类别： - 视觉问答 标签： - 鲁棒性 - LLaVA-Bench - COCO - 扰动 - Vittle - 联合扰动 美观名称："Vittle——联合扰动版LLaVA-Bench-COCO" 样本规模类别： - n<1K # Vittle——联合扰动版LLaVA-Bench-COCO 本数据集提供了**联合（视觉+文本）扰动**变体的[LLaVA-Bench（COCO）](https://arxiv.org/abs/2304.08485)开放式视觉问答基准。本数据集作为[Vittle（视觉指令瓶颈调优，Visual Instruction Bottleneck Tuning）](https://arxiv.org/abs/2505.13946)项目（NeurIPS 2025）的组成部分发布。 ## 概览 - **问题集**：90个基础问题 × 3种文本扰动变体（翻译版本） - **图像集**：30张独特的COCO val2014图像 × 3种视觉扰动变体 - **联合组合**：3种视觉扰动 × 3种文本扰动 = 9种联合扰动设置 ## 扰动组合 ### 视觉扰动（强度等级3） 基于[MM-Robustness](https://github.com/Jielin-Qiu/MM_Robustness)生成： - 高斯噪声（`COCO_IP_gaussian_noise_3`） - 变焦模糊（`COCO_IP_zoom_blur_3`） - 霜冻效果（`COCO_IP_frost_3`） ### 文本扰动（通过GPT-4o翻译） - 印地语（`qa90_questions_Hindi.jsonl`） - 希腊语（`qa90_questions_Greek.jsonl`） - 阿拉伯语（`qa90_questions_Arabic.jsonl`） ### 全部9种联合设置 | 视觉扰动 文本扰动 | 印地语 | 希腊语 | 阿拉伯语 | |---|---|---|---| | 高斯噪声 | 是 | 是 | 是 | | 变焦模糊 | 是 | 是 | 是 | | 霜冻效果 | 是 | 是 | 是 | ## 文件结构 . ├── README.md ├── qa90_questions.jsonl # 90条原始（干净）问题 ├── questions_perturbed/ │ ├── qa90_questions_Hindi.jsonl │ ├── qa90_questions_Greek.jsonl │ └── qa90_questions_Arabic.jsonl └── images/ ├── COCO_IP_gaussian_noise_3/ # 30张图像 ├── COCO_IP_zoom_blur_3/ └── COCO_IP_frost_3/ ## 使用方法 若要在某一联合扰动设置下评估模型，请将扰动后的问题文件与扰动后的图像文件夹配对。例如，针对（霜冻效果 + 印地语）场景： - 问题文件：`questions_perturbed/qa90_questions_Hindi.jsonl` - 图像文件夹：`images/COCO_IP_frost_3/` ## 引用 bibtex @inproceedings{ oh2025visual, title={Visual Instruction Bottleneck Tuning}, author={Changdae Oh and Jiatong Li and Shawn Im and Sharon Li}, booktitle={第三十九届神经信息处理系统年度会议}, year={2025}, url={https://openreview.net/forum?id=yzHiEmLSk8} } ## 许可证 MIT