MM-Hallu/vqav2-idk

---
language:
- en
license: cc-by-4.0
task_categories:
- visual-question-answering
tags:
- hallucination
- vqa
- benchmark
size_categories:
- 10K<n<100K
dataset_info:
features:
- name: image
dtype: image
- name: question_id
dtype: string
- name: question
dtype: string
- name: answer
sequence: string
- name: keywords
sequence: string
splits:
- name: train
- name: val
configs:
- config_name: default
data_files:
- split: train
path: train-*.parquet
- split: val
path: val-*.parquet
---

# VQAv2-IDK

VQAv2-IDK is a hallucination evaluation benchmark derived from the VQAv2 dataset. It consists of unanswerable (hallucination-inducing) image-question pairs where the desired answer is "I Don't Know."

## Dataset Description

- **Paper:** [Visually Dehallucinative Instruction Generation: Know What You Don't Know](https://arxiv.org/abs/2402.09717)
- **Repository:** [https://github.com/ncsoft/idk](https://github.com/ncsoft/idk)

## Dataset Structure

- **train:** 13,807 examples
- **val:** 6,624 examples

Each example contains:
- `image`: The input image
- `question_id`: Unique question identifier
- `question`: The text of the question
- `answer`: List of human-provided answers
- `keywords`: Keywords indicating unanswerability (e.g., "unknown", "none")

## Citation

```bibtex
@inproceedings{cha2024visually,
title={Visually Dehallucinative Instruction Generation: Know What You Don't Know},
author={Cha, Sungguk and Lee, Jusung and Lee, Younghyun and Yang, Cheoljong},
year={2024},
}
```