TAPOS

当前的动作识别方法主要依靠深度卷积网络来获得视觉和运动特征的特征嵌入。虽然这些方法在标准基准测试中表现出显着的性能，但我们仍然需要更好地理解视频，特别是它们的内部结构，如何与高级语义相关，这可能会带来多方面的好处，例如可解释性预测甚至新方法可以将识别性能提升到一个新的水平。为实现这一目标，我们构建了 TAPOS，这是一个在运动视频上开发的新数据集，带有子动作的手动注释，并在顶部进行时间动作解析研究。我们的研究表明，体育活动通常由多个子动作组成，对这种时间结构的认识有利于动作识别。我们还研究了许多时间解析方法，并因此设计了一种改进的方法，该方法能够在不知道它们的标签的情况下从训练数据中挖掘子动作。在构建的 TAPOS 上，所提出的方法被证明可以揭示动作内信息，即动作实例是如何由子动作组成的，以及交互动作信息，即一个特定的子动作可能通常出现在各种动作中。

Current action recognition methods predominantly rely on deep convolutional networks to derive feature embeddings of visual and motion features. While these methods have achieved remarkable performance on standard benchmark datasets, we still need a deeper understanding of how videos—especially their internal structures—correlate with high-level semantics, which could bring multifaceted benefits such as interpretable prediction, and even enable new approaches to elevate recognition performance to a new level. To achieve this research goal, we construct TAPOS, a novel dataset developed on motion videos with manual annotations for sub-actions, accompanied by corresponding temporal action parsing research. Our findings indicate that sports activities are generally composed of multiple sub-actions, and awareness of such temporal structures is beneficial to action recognition. We also investigate a variety of temporal parsing methods, and accordingly design an improved approach that can mine sub-actions from training data without requiring their labels. Evaluated on the constructed TAPOS dataset, the proposed method is validated to uncover two types of information inherent to actions: the compositional mechanism by which an action instance is composed of sub-actions, and interactive action information, meaning that a specific sub-action often appears across a wide range of actions.

搜集汇总

背景与挑战

背景概述

TAPOS是一个运动视频数据集，包含21种运动类别的子动作注释，旨在研究动作识别和时间动作解析。该数据集由香港中文大学于2020年发布，支持对视频内部结构与高级语义关系的深入理解。