SceneEgo

Egocentric 3D human pose estimation with a single head-mounted fisheye camera has recently attracted attention due to its numerous applications in virtual and augmented reality. Existing methods still struggle in challenging poses where the human body is highly occluded or is closely interacting with the scene. To address this issue, we propose a scene-aware egocentric pose estimation method that guides the prediction of the egocentric pose with scene constraints. To this end, we propose an egocentric depth estimation network to predict the scene depth map from a wide-view egocentric fisheye camera while mitigating the occlusion of the human body with a depth-inpainting network. Next, we propose a scene-aware pose estimation network that projects the 2D image features and estimated depth map of the scene into a voxel space and regresses the 3D pose with a V2V network. The voxel-based feature representation provides the direct geometric connection between 2D image features and scene geometry, and further facilitates the V2V network to constrain the predicted pose based on the estimated scene geometry. To enable the training of the aforementioned networks, we also generated a synthetic dataset, called EgoGTA, and an in-the-wild dataset based on EgoPW, called EgoPW-Scene. The experimental results of our new evaluation sequences show that the predicted 3D egocentric poses are accurate and physically plausible in terms of human-scene interaction, demonstrating that our method outperforms the state-of-the-art methods both quantitatively and qualitatively.基于单台头戴式鱼眼摄像头的第一人称视角三维人体姿态估计（egocentric 3D human pose estimation），近年来因其在虚拟现实（virtual reality）与增强现实（augmented reality）领域的诸多应用而受到广泛关注。现有方法在人体严重遮挡或与场景紧密交互的高难度姿态场景中仍表现欠佳。为解决上述问题，本文提出一种场景感知的第一人称视角姿态估计方法，通过场景约束引导第一人称姿态的预测。为此，本文提出一种第一人称视角深度估计网络，可从宽视场第一人称鱼眼摄像头中预测场景深度图，并通过深度修复网络（depth-inpainting network）缓解人体遮挡带来的干扰。随后，本文提出一种场景感知的姿态估计网络，将二维图像特征与场景估计深度图投影至体素空间（voxel space），并通过体素到体素网络（V2V network）回归得到三维姿态。基于体素的特征表征可建立二维图像特征与场景几何结构之间的直接几何关联，进一步辅助体素到体素网络基于估计得到的场景几何结构对预测姿态进行约束。为实现上述网络的训练，本文还生成了名为EgoGTA的合成数据集，以及基于EgoPW的野外真实数据集EgoPW-Scene。本文新增评估序列的实验结果表明，所预测的第一人称视角三维姿态在人体与场景交互方面兼具准确性与物理合理性，证明本文方法在定量与定性层面均优于当前最优（state-of-the-art）方法。