Alibaba-NLP/xvbench

XVBench是一个用于评估多模态检索增强生成系统在跨视频理解任务上的基准数据集，由论文《VimRAG: Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph》引入。该数据集基于HowTo100M视频库构建，HowTo100M是一个大规模叙述性教学视频集合。数据集专注于需要模型或代理从大型视频库中检索和推理视频证据的问题，与单视频问答相比，旨在测试系统是否能找到相关视觉片段、保留细粒度视觉细节，并回答依赖于跨视频片段分布信息的问题。每个数据示例包含一个问题、一个真实答案、源视频标识符以及一个或多个支持答案的参考视频片段。任务为开放式问答，语言为英语，许可证为CC BY 4.0。

XVBench is a benchmark for evaluating multimodal retrieval-augmented generation systems on cross-video understanding, introduced alongside the paper VimRAG: Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph. The questions are created based on videos from HowTo100M, a large-scale corpus of narrated instructional videos. The benchmark focuses on questions that require models or agents to retrieve and reason over video evidence from this large video corpus, testing whether a system can find relevant visual clips, preserve fine-grained visual details, and answer questions depending on information distributed across video segments. Each example contains a question, a ground-truth answer, the source video identifier, and one or more reference video clips that support the answer. The task is open-ended question answering, the language is English, and the license is CC BY 4.0.