MSRVTT

我们介绍了msr-vtt (代表 “MSR视频到文本”)，它是一种新的视频理解的大规模视频基准，尤其是将视频转换为文本的新兴任务。这是通过从商业视频搜索引擎收集257个流行的查询来实现的，每个查询都有118个视频。在当前版本中，msr-vtt提供10k网络视频剪辑，总共41.2小时，200K个剪辑句子对，涵盖了最全面的类别和多样化的视觉内容，并代表了句子和词汇方面的最大数据集。每个片段由1,327名AMT工作者用大约20个自然句子进行注释。与现有数据集的完整集合相比，我们对msr-vtt进行了详细分析，并总结了不同的最新视频到文本方法。我们还在此数据集上对这些方法进行了广泛的评估，表明基于混合递归神经网络的方法将单帧和运动表示与软注意力池策略相结合，在msr-vtt上产生了最佳的泛化能力。

We introduce MSR-VTT (short for "Microsoft Research Video-to-Text"), a large-scale video benchmark for video understanding, especially the emerging task of video-to-text generation. This is achieved by collecting 257 popular queries from a commercial video search engine, with 118 videos per query. In its current version, MSR-VTT provides 10,000 online video clips totaling 41.2 hours, along with 200,000 clip-sentence pairs. It covers the most comprehensive categories and diverse visual content, and stands as the largest dataset in terms of sentences and vocabulary. Each clip is annotated with approximately 20 natural sentences by 1,327 AMT workers. Compared with the complete set of existing datasets, we conduct a detailed analysis of MSR-VTT and summarize different state-of-the-art video-to-text methods. We also perform extensive evaluations of these methods on this dataset, demonstrating that methods based on hybrid recurrent neural networks, which combine single-frame and motion representations with soft attention pooling strategies, yield the best generalization performance on MSR-VTT.

搜集汇总

背景与挑战

背景概述

MSRVTT是一个由Microsoft Research于2016年发布的大规模视频描述数据集，包含10,000个网络视频剪辑（总计41.2小时）和200,000个剪辑句子对，每个视频由人工标注约20个自然句子，覆盖257个查询和多样化的视觉内容。该数据集专为视频到文本任务设计，如视频描述生成，是目前该领域最全面的基准之一，支持广泛的视频理解研究。