CapQA
收藏资源简介：
CapQA数据集由爱笔科技和清华大学的研究团队创建，旨在支持复杂视觉推理和问答任务的研究。该数据集包含1000张细粒度活动图像，数据来源于Consented Activities of People (CAP)数据集，经过筛选和标注生成。数据集的创建过程包括从CAP数据集中选择20种活动，每种活动随机提取50个视频片段，并从每个片段中选择一个关键帧作为图像数据。随后，使用GPT-4v自动生成包括问题、答案、详细描述和总结性标题的标注信息。CapQA数据集的应用领域主要集中在视觉推理和问答任务中，旨在减少幻觉现象并提升模型对细粒度图像细节的描述能力。
The CapQA dataset was developed by research teams from Aibee Technology and Tsinghua University, with the purpose of supporting research on complex visual reasoning and question answering (QA) tasks. It contains 1000 fine-grained activity images sourced from the Consented Activities of People (CAP) dataset, which were generated through filtering and annotation processes. The dataset creation workflow includes selecting 20 activity categories from the CAP dataset, randomly extracting 50 video clips for each category, and selecting one key frame from each clip as the image data. Subsequently, GPT-4V is used to automatically generate annotation information including questions, answers, detailed descriptions and summary titles. The CapQA dataset is primarily applied in visual reasoning and QA tasks, aiming to reduce hallucinations and enhance models' capability to describe fine-grained image details.
数据集概述
数据集名称
Socratic Questioning: Learn to Self-guide Multimodal Reasoning in the Wild
数据集来源
该数据集是论文《Socratic Questioning: Learn to Self-guide Multimodal Reasoning in the Wild》的官方实现，数据集可通过CapQA获取。
数据集用途
该数据集用于多模态推理任务，支持单轮和三轮推理评估。
数据集内容
数据集包含以下三个子数据集：
-
ScienceQA
- 数据文件：
images,pid_splits.json,problems.json - 下载地址：ScienceQA repo
- 评估脚本：
- 单轮推理：
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/sqa.sh - 三轮推理：
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/sqa_3turn.sh
- 单轮推理：
- 数据文件：
-
TextVQA
- 数据文件：
TextVQA_0.5.1_val.json,train_val_images.zip - 下载地址：
TextVQA_0.5.1_val.json：TextVQA_0.5.1_val.jsontrain_val_images.zip：train_val_images.zip
- 评估脚本：
- 单轮推理：
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/textvqa.sh - 三轮推理：
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/textvqa_3turn.sh
- 单轮推理：
- 数据文件：
-
MM-Vet
- 数据文件：
mm-vet.zip - 下载地址：mm-vet.zip
- 评估脚本：
- 单轮推理：
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mmvet.sh - 三轮推理：
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mmvet_3turn.sh
- 单轮推理：
- 评估方法：使用官方Jupyter Notebook进行评估，具体方法参考MM-Vet。
- 数据文件：
数据集使用说明
- 1Socratic Questioning: Learn to Self-guide Multimodal Reasoning in the Wild爱笔科技, 清华大学 · 2025年
