lccshunli/MVBench

MVBench是一个视频理解基准数据集，采用一种新颖的静态到动态方法来定义时间相关任务。通过将静态任务转换为动态任务，系统生成需要从感知到认知广泛时间能力的视频任务。基于任务定义，自动将公共视频注释转换为多项选择QA进行评估。这种范式以最小人工干预高效创建数据集，并通过真实视频注释确保评估公平性。数据集涵盖20个时间任务示例，用于评估多模态大语言模型的时间推理能力。

MVBench is a video understanding benchmark dataset that introduces a novel static-to-dynamic method for defining temporal-related tasks. By converting static tasks into dynamic ones, it facilitates systematic generation of video tasks requiring a wide range of temporal abilities from perception to cognition. Guided by task definitions, public video annotations are automatically transformed into multiple-choice QA for evaluation. This paradigm enables efficient creation with minimal manual intervention while ensuring evaluation fairness through ground-truth video annotations. The dataset includes 20 temporal task examples for assessing temporal reasoning in multimodal large language models.