Replication Data for: Summarizing First-Person Videos from Third Persons' Points of Views

Video highlight or summarization is among interesting topics in computer vision, which benefits a variety of applications like viewing, searching, or storage. However, most existing studies rely on training data of third-person videos, which cannot easily generalize to highlight the first-person ones. With the goal of deriving an effective model to summarize first-person videos, we propose a novel deep neural network architecture for describing and discriminating vital spatiotemporal information across videos with different points of view. Our proposed model is realized in a semi-supervised setting, in which fully annotated third-person videos, unlabeled first-person videos, and a small number of annotated first-person ones are presented during training. In our experiments, qualitative and quantitative evaluations on both benchmarks and our collected first-person video datasets are presented.

视频高光提取与摘要生成是计算机视觉领域的热门研究方向之一，可为视频浏览、内容检索与存储等诸多应用场景提供助力。然而现有多数研究均以第三人称视角视频作为训练数据，此类模型难以直接泛化适配第一人称视角视频的高光提取任务。为构建可有效完成第一人称视角视频摘要生成的模型，我们提出一种新颖的深度神经网络（deep neural network）架构，用于描述并区分不同视角视频中的关键时空信息。所提模型采用半监督训练范式，训练过程中可利用全标注第三人称视角视频、未标注第一人称视角视频，以及少量标注第一人称视角视频作为训练数据。实验部分我们在公开基准数据集与本团队采集的第一人称视角视频数据集上，分别开展了定性与定量评估。