DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenes

DisasterVQA is a benchmark dataset for evaluating Vision-Language Models (VLMs) on disaster-response visual question answering. It contains 1,395 real-world disaster images and 4,405 expert-curated question–answer pairs covering floods, wildfires, and earthquakes, sourced from three public datasets (MEDIC, CrisisMMD, Incidents1M).

The dataset includes three question types:

Binary (Yes/No)

Multiple-Choice

Open-Ended

Questions span situational awareness and operational decision-making tasks, grounded in humanitarian frameworks (FEMA ESF, OCHA MIRA). We benchmark seven state-of-the-art vision–language models, revealing performance gaps in fine-grained quantitative reasoning, object counting, and context-sensitive interpretation — especially for underrepresented disaster scenarios.

Files:

disastervqa_annotations.jsonl: Benchmark annotations and metadata (question text, ground-truth answers, image paths, and taxonomy labels)

disastervqa_model_outputs.jsonl: Model predictions for each question (join with annotations using question_id). Open-Ended questions may include a judge-LLM decision label (Right/Wrong)

taxonomy.json: Final taxonomy definitions and references for each crisis_info_code

License: CC BY-SA 4.0

Paper: "DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenes", ICWSM 2026. arXiv:2601.13839

HuggingFace Dataset: https://huggingface.co/datasets/QCRI/DisasterVQA

DisasterVQA是一款用于评估视觉语言模型（Vision-Language Models，VLMs）在灾害响应视觉问答任务上性能的基准数据集。该数据集包含1395张真实灾害场景图像，以及4405组专家精心编撰的问答对，覆盖洪水、野火与地震三类灾害场景，数据来源于MEDIC、CrisisMMD与Incidents1M三个公开数据集。

该数据集包含三类问题类型：二元（是/否）类、多项选择类与开放问答类。

问题涵盖态势感知与行动决策两类任务，其设计基于人道主义行动框架（美国联邦应急管理局应急支持功能FEMA ESF、联合国人道主义事务协调厅灾害影响快速评估OCHA MIRA）。我们针对7款当前主流顶尖的视觉语言模型开展了基准评测，结果表明，这些模型在细粒度定量推理、目标计数与上下文敏感解读等方面存在明显性能缺口，尤其在代表性不足的灾害场景中表现尤为突出。

文件说明：
disastervqa_annotations.jsonl：基准数据集标注与元数据，包含问题文本、标准答案、图像路径与分类标签
disastervqa_model_outputs.jsonl：各问题的模型预测结果，可通过question_id与标注文件关联；开放问答类问题可能包含评审大语言模型给出的判定标签（正确/错误）
taxonomy.json：各类危机信息编码的最终分类体系定义与参考资料

许可证：CC BY-SA 4.0（知识共享署名-相同方式共享4.0协议）

相关论文：《DisasterVQA：面向灾害场景的视觉问答基准数据集》，发表于ICWSM 2026。arXiv:2601.13839

HuggingFace数据集链接：https://huggingface.co/datasets/QCRI/DisasterVQA