HyperAIHyperAI
2 months ago

Scene-aware Egocentric 3D Human Pose Estimation

Wang, Jian ; Liu, Lingjie ; Xu, Weipeng ; Sarkar, Kripasindhu ; Luvizon, Diogo ; Theobalt, Christian
Scene-aware Egocentric 3D Human Pose Estimation
Abstract

Egocentric 3D human pose estimation with a single head-mounted fisheye camerahas recently attracted attention due to its numerous applications in virtualand augmented reality. Existing methods still struggle in challenging poseswhere the human body is highly occluded or is closely interacting with thescene. To address this issue, we propose a scene-aware egocentric poseestimation method that guides the prediction of the egocentric pose with sceneconstraints. To this end, we propose an egocentric depth estimation network topredict the scene depth map from a wide-view egocentric fisheye camera whilemitigating the occlusion of the human body with a depth-inpainting network.Next, we propose a scene-aware pose estimation network that projects the 2Dimage features and estimated depth map of the scene into a voxel space andregresses the 3D pose with a V2V network. The voxel-based featurerepresentation provides the direct geometric connection between 2D imagefeatures and scene geometry, and further facilitates the V2V network toconstrain the predicted pose based on the estimated scene geometry. To enablethe training of the aforementioned networks, we also generated a syntheticdataset, called EgoGTA, and an in-the-wild dataset based on EgoPW, calledEgoPW-Scene. The experimental results of our new evaluation sequences show thatthe predicted 3D egocentric poses are accurate and physically plausible interms of human-scene interaction, demonstrating that our method outperforms thestate-of-the-art methods both quantitatively and qualitatively.

Scene-aware Egocentric 3D Human Pose Estimation | Latest Papers | HyperAI