Weizmann Dataset 人体行为动作形状的数据集

概述：2005年，以色列 Weizmann institute 发布了Weizmann 数据库。数据库包含了 10个动作（bend, jack, jump, pjump, run,side, skip, walk, wave1,wave2），每个动作有 9 个不同的样本。视频的视角是固定的，背景相对简单，每一帧中只有 1 个人做动作。 数据库中标注数据除了类别标记外还包括：前景的行为人剪影和用于背景抽取的背景序列。 Human action in video sequences can be seen as silhouettes of a moving torso and protruding limbs undergoing articulated motion. We regard human actions as three-dimensional shapes induced by the silhouettes in the space-time volume. We adopt a recent approach by Gorelick et. al. for analyzing 2D shapes and generalize it to deal with volumetric space-time action shapes. Our method utilizes properties of the solution to the Poisson equation to extract space-time features such as local space-time saliency, action dynamics, shape structure and orientation. We show that these features are useful for action recognition, detection and clustering. The method is fast, does not require video alignment and is applicable in (but not limited to) many scenarios where the background is known. Moreover, we demonstrate the robustness of our method to partial occlusions, non-rigid deformations, significant changes in scale and viewpoint, high irregularities in the performance of an action and low quality video. NEW! The PAMI paper (full version, updated results) in PDF (2MB) format (BibTeX). Updated database - including original silhouette sequences and their aligned version, as well as the robustness sequences, can be found below. The ICCV paper (shorter version) in PDF (2MB) format (BibTeX). We use the solution of the Poisson equation to extract several space time features. In the table below we demonstrate these features for three sequences of different actions. The first two columns show the original video sequence and the extracted foreground mask. The third column shows the solutions of the Poisson equation, color-coded from blue (low values) to red (high values). The last three columns show the Space-Time ''saliency'', ''plateness'' and ''stickness'' features that we use. See the paper for details. Click the images below to play the full video sequences. In the paper we report results for four experiments: action clustering, action recognition, robustness experiments and action detection. Here we show results of last three. We collected a database of 90 low-resolution (180 x 144, deinterlaced 50 fps) video sequences showing nine different people, each performing 10 natural actions such as run,walk,skip,jumping-jack(or shortly jack), jump-forward-on-two-legs (or jump), jump-in-place-on-two-legs(or pjump), gallopsideways(or side), wave-two-hands (or wave2), waveone- hand (or wave1), or bend. In order to treat both the periodic and nonperiodic actions in the same framework as well as to compensate for different length of periods, we used a sliding window in time to extract space-time cubes, each having eight frames with an overlap of four frames between the consecutive space-time cubes. Below we summarize our recognition rates in "leave-one-sequence-out" classification experiments for both complete sequences and sub-sequences . In this experiment we demonstrate the robustness of our method to high irregularities in the performance of an action. We collected ten test video sequences of people walking in various difficult scenarios in front of different non-uniform backgrounds (see the sequences and their foreground masks below). We show that our approach has relatively low sensitivity to partial occlusions, non-rigid deformations and other defects in the extracted space-time shape. Click the images below to play the full video sequences. Experiment results: The table below shows for each of the test sequences the first and second best choices and their distances as well as the median distance to all the actions in our database. The test sequences are sorted by the distance to their first best chosen action. All the sequences were classified as "walk". Moreover we demonstrate the robustness of our method to substential changes in viewpoint. For this purpose we collected ten additional sequences, each showing the "walk" action captured from a different viewpoint (varying between 0° and 81° relative to the image plane with steps of 9°). Note, that sequences with angles approaching 90 degrees contain significant changes in scale within the sequence. All sequences with viewpoints between 0° and 54° were classified correctly with a large relative gap between the first (true) and the second closest actions (see table below). For larger viewpoints a gradual deterioration occurs. This demonstrates the robustness of our method to relatively large variations in viewpoint. This experiment shows action detection on a movie sequence of a ballet dance, performed by the "Birmingham Royal Ballet" from the "London Dance" website. Original full video can be found also here (WMV format, 400KB). The task was to detect all instances of the ''cabriole'' pa (the query) in the input video. Click the images below to play the full video sequences. The PAMI paper:

概述：2005年，以色列魏茨曼科学研究所（Weizmann Institute）发布了魏茨曼数据集（Weizmann Database）。该数据集包含10类动作：弯腰（bend）、开合跳（jack）、跳跃（jump）、双腿前跳（pjump）、跑步（run）、侧跑（side）、跳跃行进（skip）、行走（walk）、单臂挥动（wave1）、双臂挥动（wave2），每类动作对应9个不同样本。视频采用固定视角拍摄，背景相对简洁，每一帧仅包含一名正在执行动作的人物。数据集标注除类别标签外，还包含前景行人剪影以及用于背景提取的背景序列。

视频序列中的人体动作可被视为由运动躯干与做关节运动的突出肢体所构成的剪影。我们将人体动作视作时空体中由剪影所诱导出的三维形状。我们采用Gorelick等人近期提出的二维形状分析方法，并将其推广至体素时空动作形状的处理任务。我们的方法利用泊松方程（Poisson equation）解的特性，提取局部时空显著性、动作动态、形状结构与方向等时空特征。实验证明，此类特征可有效应用于动作识别、检测与聚类任务。该方法运算速度快，无需视频对齐，且可适用于（但不限于）背景已知的多种场景。此外，我们验证了该方法对部分遮挡、非刚性形变、尺度与视角的显著变化、动作执行的高度不规则性以及低质量视频均具备良好鲁棒性。

【新增内容】IEEE模式分析与机器智能汇刊（PAMI）论文（完整版，含更新结果，PDF格式，大小2MB，附BibTeX引用格式）。更新后的数据集（包含原始剪影序列及其对齐版本，以及鲁棒性测试序列）可在下方获取。国际计算机视觉大会（ICCV）论文（精简版，PDF格式，大小2MB，附BibTeX引用格式）。

我们通过求解泊松方程提取若干时空特征。下表展示了三类不同动作序列的上述特征：前两列分别为原始视频序列与提取得到的前景掩码；第三列为泊松方程的解，采用从蓝色（低值）到红色（高值）的颜色编码；最后三列则为我们所使用的时空「显著性」「板状性（plateness）」与「杆状性（stickness）」特征。详细内容请参阅论文。点击下方图片可播放完整视频序列。

本文中我们报告了四类实验的结果：动作聚类、动作识别、鲁棒性测试与动作检测。下文将展示后三类实验的结果。我们构建了一个包含90个低分辨率（180×144，去隔行处理，帧率50fps）视频序列的数据集，涉及9名不同受试者，每名受试者完成10种日常动作：跑步、行走、跳跃行进、开合跳（简称jack）、双腿前跳、原地双腿跳（简称pjump）、侧跑（简称side）、双臂挥动（简称wave2）、单臂挥动（简称wave1）以及弯腰。为了在统一框架下处理周期性与非周期性动作，并补偿不同动作周期的长度差异，我们采用时间滑动窗口提取时空立方体：每个时空立方体包含8帧，相邻时空立方体之间存在4帧的重叠。下文将总结我们在「留一序列交叉验证」分类实验中针对完整序列与子序列的识别准确率。

本实验旨在验证我们的方法对动作执行高度不规则性的鲁棒性。我们采集了10段测试视频序列，内容为受试者在多种复杂场景下、于不同非均匀背景前行走的画面（相关序列及其前景掩码可参见下方内容）。实验证明，我们的方法对部分遮挡、非刚性形变以及提取得到的时空形状中的其他缺陷均具备较低的敏感性。点击下方图片可播放完整视频序列。

实验结果：下表展示了每段测试序列的前两个最优匹配类别及其距离，以及与数据集中所有动作的距离中位数。测试序列按照其与第一最优匹配类别的距离进行排序。所有测试序列均被分类为「行走」。此外，我们验证了方法对视角显著变化的鲁棒性。为此我们额外采集了10段序列，每段均为从不同视角拍摄的「行走」动作（相对于成像平面的视角范围为0°至81°，步长为9°）。需要注意的是，视角接近90°的序列内部存在显著的尺度变化。所有视角在0°至54°之间的序列均被正确分类，且第一匹配（真实类别）与第二匹配类别的相对差距较大（详见下表）。当视角进一步增大时，分类准确率会逐渐下降。这证明了我们的方法对视角的较大幅度变化具备良好鲁棒性。

本实验展示了针对一段芭蕾舞电影序列的动作检测任务，该序列由伦敦舞蹈网站（London Dance）上的伯明翰皇家芭蕾舞团（Birmingham Royal Ballet）表演。完整原始视频可在此处获取（WMV格式，大小400KB）。任务目标为在输入视频中检测出所有「卡布里奥尔舞步（cabriole）」（查询样本）的出现实例。点击下方图片可播放完整视频序列。

PAMI论文：

搜集汇总

背景与挑战

背景概述

Weizmann Dataset是一个专注于人体行为动作的数据集，包含10种动作的90个样本，适用于动作识别和检测研究。其特点是固定视角、简单背景和丰富的标注信息，包括行为人的剪影和背景序列。