xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera

We present a new solution to egocentric 3D body pose estimation frommonocular images captured from a downward looking fish-eye camera installed onthe rim of a head mounted virtual reality device. This unusual viewpoint, just2 cm. away from the user's face, leads to images with unique visual appearance,characterized by severe self-occlusions and strong perspective distortions thatresult in a drastic difference in resolution between lower and upper body. Ourcontribution is two-fold. Firstly, we propose a new encoder-decoderarchitecture with a novel dual branch decoder designed specifically to accountfor the varying uncertainty in the 2D joint locations. Our quantitativeevaluation, both on synthetic and real-world datasets, shows that our strategyleads to substantial improvements in accuracy over state of the art egocentricpose estimation approaches. Our second contribution is a new large-scalephotorealistic synthetic dataset - xR-EgoPose - offering 383K frames of highquality renderings of people with a diversity of skin tones, body shapes,clothing, in a variety of backgrounds and lighting conditions, performing arange of actions. Our experiments show that the high variability in our newsynthetic training corpus leads to good generalization to real world footageand to state of the art results on real world datasets with ground truth.Moreover, an evaluation on the Human3.6M benchmark shows that the performanceof our method is on par with top performing approaches on the more classicproblem of 3D human pose from a third person viewpoint.