Violin (VIdeO-and-Language INference)

视频和语言推理是对视频和文本进行联合多模态理解的任务。给定一个以对齐字幕为前提的视频剪辑，再加上基于视频内容的自然语言假设，模型需要推断该假设是否与给定的视频剪辑相矛盾。 Violin 数据集是该任务的数据集，包含来自 15,887 个视频剪辑的 95,322 个视频-假设对，跨越超过 582 小时的视频。这些视频剪辑包含丰富的内容，具有不同的时间动态、事件变化和人际互动，从两个来源收集：(i) 流行的电视节目，以及 (ii) 来自 YouTube 频道的电影剪辑。

Video-and-language reasoning is a joint multimodal comprehension task that focuses on unified understanding of video and textual content. Given a video clip paired with its aligned subtitles as the premise, alongside a natural language hypothesis derived from the video's content, models are required to infer whether the hypothesis contradicts the given video clip. The Violin dataset is a benchmark dataset for this task, comprising 95,322 video-hypothesis pairs from 15,887 video clips, spanning over 582 hours of video content. These video clips feature rich and diverse content with varied temporal dynamics, event changes, and interpersonal interactions, and are collected from two sources: (i) popular television programs, and (ii) movie clips from YouTube channels.

搜集汇总

背景与挑战

背景概述

Violin数据集是一个用于视频与语言推理任务的大规模数据集，包含95,322个视频-假设对，源自15,887个视频剪辑，覆盖超过582小时视频，内容来自电视节目和YouTube电影剪辑。其核心任务是判断自然语言假设是否与给定视频内容相矛盾。