changdae/vittle-llavabench-coco-textual-perturbed

--- license: mit task_categories: - visual-question-answering tags: - robustness - LLaVA-Bench - COCO - perturbation - vittle - text-perturbation pretty_name: "Vittle - Textually Perturbed LLaVA-Bench-COCO" size_categories: - n<1K --- # Vittle - Textually Perturbed LLaVA-Bench-COCO This dataset provides **textually perturbed** variants of the [LLaVA-Bench (COCO)](https://arxiv.org/abs/2304.08485) open-ended VQA benchmark. It is released as part of the [Vittle (Visual Instruction Bottleneck Tuning)](https://arxiv.org/abs/2505.13946) project (NeurIPS 2025). ## Overview - **Questions**: 90 base questions x 9 textual perturbation variants = 810 perturbed questions. Clean images are used. - **Images**: 30 unique COCO val2014 images (clean, unperturbed) ## Textual Perturbations Generated following [MM-Robustness](https://github.com/Jielin-Qiu/MM_Robustness) for char/word-level, and GPT-4o for sentence-level (translation): ### Char/Word-level Perturbations | Perturbation | File | Description | |---|---|---| | Random Delete | `qa90_questions_rd_7.jsonl` | Random character deletion (severity 7) | | Random Swap | `qa90_questions_rs_4.jsonl` | Random character swap (severity 4) | | Random Insert | `qa90_questions_ri_4.jsonl` | Random character insertion (severity 4) | | Keyboard Aug | `qa90_questions_KeyboardAug_3.jsonl` | Keyboard-based typo augmentation (severity 3) | | Char Delete | `qa90_questions_RandomCharAug_delete_3.jsonl` | Random character deletion augmentation (severity 3) | | Char Insert | `qa90_questions_RandomCharAug_insert_3.jsonl` | Random character insertion augmentation (severity 3) | ### Sentence-level Perturbations (Translation) | Perturbation | File | Description | |---|---|---| | Hindi | `qa90_questions_Hindi.jsonl` | GPT-4o translation to Hindi | | Greek | `qa90_questions_Greek.jsonl` | GPT-4o translation to Greek | | Arabic | `qa90_questions_Arabic.jsonl` | GPT-4o translation to Arabic | ## File Structure ``` . ├── README.md ├── qa90_questions.jsonl # 90 original (clean) questions ├── questions_perturbed/ │ ├── qa90_questions_rd_7.jsonl │ ├── qa90_questions_rs_4.jsonl │ ├── qa90_questions_ri_4.jsonl │ ├── qa90_questions_KeyboardAug_3.jsonl │ ├── qa90_questions_RandomCharAug_delete_3.jsonl │ ├── qa90_questions_RandomCharAug_insert_3.jsonl │ ├── qa90_questions_Hindi.jsonl │ ├── qa90_questions_Greek.jsonl │ └── qa90_questions_Arabic.jsonl └── images/ └── val2014/ # 30 clean COCO images ``` ## Citation ```bibtex @inproceedings{ oh2025visual, title={Visual Instruction Bottleneck Tuning}, author={Changdae Oh and Jiatong Li and Shawn Im and Sharon Li}, booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems}, year={2025}, url={https://openreview.net/forum?id=yzHiEmLSk8} } ``` ## License MIT许可证：MIT 任务类别： - 视觉问答（Visual Question Answering，VQA） 标签： - 鲁棒性 - LLaVA-Bench - COCO - 扰动 - Vittle - 文本扰动 美观名称："Vittle——文本扰动版LLaVA-Bench-COCO" 样本规模类别： - n<1K # Vittle——文本扰动版LLaVA-Bench-COCO 本数据集提供了**文本扰动变体**的[LLaVA-Bench（COCO）](https://arxiv.org/abs/2304.08485)开放式视觉问答基准。本数据集作为[Vittle（视觉指令瓶颈调优，Visual Instruction Bottleneck Tuning）](https://arxiv.org/abs/2505.13946)项目（NeurIPS 2025）的一部分发布。 ## 概览 - **问题集**：90个基础问题 × 9种文本扰动变体 = 810个扰动问题。实验采用干净图像作为输入。 - **图像集**：30张独特的COCO val2014数据集干净未扰动图像。 ## 文本扰动方法 文本扰动生成遵循[MM-Robustness](https://github.com/Jielin-Qiu/MM_Robustness)实现字符/词级扰动，采用GPT-4o实现句子级（翻译类）扰动： ### 字符/词级扰动 | 扰动类型 | 文件名 | 描述 | |---|---|---| | 随机删除 | `qa90_questions_rd_7.jsonl` | 随机字符删除（扰动强度7） | | 随机交换 | `qa90_questions_rs_4.jsonl` | 随机字符交换（扰动强度4） | | 随机插入 | `qa90_questions_ri_4.jsonl` | 随机字符插入（扰动强度4） | | 键盘模拟错误 | `qa90_questions_KeyboardAug_3.jsonl` | 基于键盘布局的打字错误增强（扰动强度3） | | 字符删除增强 | `qa90_questions_RandomCharAug_delete_3.jsonl` | 随机字符删除增强（扰动强度3） | | 字符插入增强 | `qa90_questions_RandomCharAug_insert_3.jsonl` | 随机字符插入增强（扰动强度3） | ### 句子级扰动（翻译类） | 扰动类型 | 文件名 | 描述 | |---|---|---| | 印地语 | `qa90_questions_Hindi.jsonl` | 通过GPT-4o翻译为印地语 | | 希腊语 | `qa90_questions_Greek.jsonl` | 通过GPT-4o翻译为希腊语 | | 阿拉伯语 | `qa90_questions_Arabic.jsonl` | 通过GPT-4o翻译为阿拉伯语 | ## 文件结构 . ├── README.md ├── qa90_questions.jsonl # 90条原始（干净）问题 ├── questions_perturbed/ │ ├── qa90_questions_rd_7.jsonl │ ├── qa90_questions_rs_4.jsonl │ ├── qa90_questions_ri_4.jsonl │ ├── qa90_questions_KeyboardAug_3.jsonl │ ├── qa90_questions_RandomCharAug_delete_3.jsonl │ ├── qa90_questions_RandomCharAug_insert_3.jsonl │ ├── qa90_questions_Hindi.jsonl │ ├── qa90_questions_Greek.jsonl │ └── qa90_questions_Arabic.jsonl └── images/ └── val2014/ # 30张干净的COCO数据集图像 ## 引用 bibtex @inproceedings{ oh2025visual, title={Visual Instruction Bottleneck Tuning}, author={Changdae Oh and Jiatong Li and Shawn Im and Sharon Li}, booktitle={第三十九届神经信息处理系统年度会议}, year={2025}, url={https://openreview.net/forum?id=yzHiEmLSk8} } ## 许可证 MIT许可证