Learning to Regress Bodies from Images using Differentiable Semantic Rendering

Learning to regress 3D human body shape and pose (e.g.~SMPL parameters) frommonocular images typically exploits losses on 2D keypoints, silhouettes, and/orpart-segmentation when 3D training data is not available. Such losses, however,are limited because 2D keypoints do not supervise body shape and segmentationsof people in clothing do not match projected minimally-clothed SMPL shapes. Toexploit richer image information about clothed people, we introducehigher-level semantic information about clothing to penalize clothed andnon-clothed regions of the image differently. To do so, we train a bodyregressor using a novel Differentiable Semantic Rendering - DSR loss. ForMinimally-Clothed regions, we define the DSR-MC loss, which encourages a tightmatch between a rendered SMPL body and the minimally-clothed regions of theimage. For clothed regions, we define the DSR-C loss to encourage the renderedSMPL body to be inside the clothing mask. To ensure end-to-end differentiabletraining, we learn a semantic clothing prior for SMPL vertices from thousandsof clothed human scans. We perform extensive qualitative and quantitativeexperiments to evaluate the role of clothing semantics on the accuracy of 3Dhuman pose and shape estimation. We outperform all previous state-of-the-artmethods on 3DPW and Human3.6M and obtain on par results on MPI-INF-3DHP. Codeand trained models are available for research at https://dsr.is.tue.mpg.de/.