DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenes

DisasterVQA is a benchmark dataset for evaluating Vision-Language Models (VLMs) on disaster-response visual question answering. It contains 1,395 real-world disaster images and 4,405 expert-curated question–answer pairs covering floods, wildfires, and earthquakes, sourced from three public datasets (MEDIC, CrisisMMD, Incidents1M).

The dataset includes three question types:

Binary (Yes/No)

Multiple-Choice

Open-Ended

Questions span situational awareness and operational decision-making tasks, grounded in humanitarian frameworks (FEMA ESF, OCHA MIRA). We benchmark seven state-of-the-art vision–language models, revealing performance gaps in fine-grained quantitative reasoning, object counting, and context-sensitive interpretation — especially for underrepresented disaster scenarios.

Files:

disastervqa_annotations.jsonl: Benchmark annotations and metadata (question text, ground-truth answers, image paths, and taxonomy labels)

disastervqa_model_outputs.jsonl: Model predictions for each question (join with annotations using question_id). Open-Ended questions may include a judge-LLM decision label (Right/Wrong)

taxonomy.json: Final taxonomy definitions and references for each crisis_info_code

License: CC BY-SA 4.0

Paper: "DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenes", ICWSM 2026. arXiv:2601.13839

HuggingFace Dataset: https://huggingface.co/datasets/QCRI/DisasterVQA