人体姿势系统 (HPS)数据集：使用车载传感器在大场景中进行 3D 人体姿势估计和自我定位

概述。HPS只使用可穿戴的传感器，联合估计大型三维场景中主体的全三维人体姿势和位置。我们使用IMU数据、头戴式摄像机的RGB视频以及预先扫描的场景作为输入。我们使用IMU数据获得一个近似的三维身体姿势，并使用头戴式摄像机的自我定位来定位三维场景中的主体。然后，我们将近似的身体姿势、摄像机的位置和方向以及三维场景进行联合优化，得到最终的位置和姿势估计。 我们介绍（HPS）人类POSEitioning系统，这是一种利用可穿戴的传感器，通过对周围环境的三维扫描来恢复人类的全部三维姿态的方法。使用连接在身体四肢的IMU和一个向外看的头戴式摄像机，HPS将基于摄像机的自我定位与基于IMU的人体跟踪结合起来。前者提供无漂移但有噪声的位置和方向估计，而后者在短期内是准确的，但在较长时间内会有漂移。我们表明，我们基于优化的整合利用了二者的优势，从而实现了无漂移的姿势精度。此外，我们还将三维场景约束纳入我们的优化，如脚与地面的接触，从而产生物理上可信的运动。HPS补充了更常见的基于第三人称的三维姿势估计方法。它允许捕捉更大的记录量和更长的运动时间，并可用于VR/AR应用，在这些应用中，人类与场景互动而不需要与外部摄像机有直接的视线，或者用于训练像真实人类一样基于第一人称视觉输入来导航和与环境互动的代理。通过HPS，我们记录了一个人类与大型3D场景（300-1000平方米）互动的数据集，其中包括7个对象和超过3小时的不同运动。 By downloading the Data, you agree to the license terms. The full dataset is available as "3D scene scans", "Videos from the head-mounted camera", "Localization results for the videos", "Original IMU poses (adopted for SMPL)" and "HPS results". The description of the files is explained in the Demo repository.

Overview. HPS uses only wearable sensors to jointly estimate the full 3D human pose and position of a subject in a large 3D scene. We take IMU data, RGB videos from a head-mounted camera, and pre-scanned 3D scenes as inputs. We use IMU data to obtain an approximate 3D body pose, and use the self-localization of the head-mounted camera to locate the subject in the 3D scene. We then perform joint optimization on the approximate body pose, the camera's position and orientation, and the 3D scene to obtain the final position and pose estimates.

We introduce the Human POSEitioning System (HPS), a method that recovers the full 3D pose of a human using wearable sensors and 3D scans of the surrounding environment. Using IMUs attached to the limbs of the body and an outward-facing head-mounted camera, HPS combines camera-based self-localization with IMU-based human tracking. The former provides drift-free but noisy position and orientation estimates, while the latter is accurate in the short term but drifts over longer periods. We show that our optimization-based integration leverages the strengths of both approaches, achieving drift-free pose accuracy. Additionally, we incorporate 3D scene constraints into our optimization, such as foot-ground contact, to produce physically plausible motion. HPS complements the more common third-person-based 3D pose estimation methods. It enables capturing larger recording volumes and longer motion durations, and can be used in VR/AR applications where humans interact with the scene without direct line-of-sight to external cameras, or for training agents that navigate and interact with environments based on first-person visual inputs like real humans.

Using HPS, we have recorded a dataset of humans interacting with large 3D scenes (300–1000 m²), which includes 7 subjects and over 3 hours of diverse motion.

By downloading the Data, you agree to the license terms. The full dataset is available as "3D scene scans", "Videos from the head-mounted camera", "Localization results for the videos", "Original IMU poses (adopted for SMPL)" and "HPS results". The description of the files is explained in the Demo repository.