Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera

We propose the first real-time approach for the egocentric estimation of 3Dhuman body pose in a wide range of unconstrained everyday activities. Thissetting has a unique set of challenges, such as mobility of the hardware setup,and robustness to long capture sessions with fast recovery from trackingfailures. We tackle these challenges based on a novel lightweight setup thatconverts a standard baseball cap to a device for high-quality pose estimationbased on a single cap-mounted fisheye camera. From the captured egocentric livestream, our CNN based 3D pose estimation approach runs at 60Hz on aconsumer-level GPU. In addition to the novel hardware setup, our other maincontributions are: 1) a large ground truth training corpus of top-down fisheyeimages and 2) a novel disentangled 3D pose estimation approach that takes theunique properties of the egocentric viewpoint into account. As shown by ourevaluation, we achieve lower 3D joint error as well as better 2D overlay thanthe existing baselines.