Unite the People: Closing the Loop Between 3D and 2D Human Representations

3D models provide a common ground for different representations of humanbodies. In turn, robust 2D estimation has proven to be a powerful tool toobtain 3D fits "in-the- wild". However, depending on the level of detail, itcan be hard to impossible to acquire labeled data for training 2D estimators onlarge scale. We propose a hybrid approach to this problem: with an extendedversion of the recently introduced SMPLify method, we obtain high quality 3Dbody model fits for multiple human pose datasets. Human annotators solely sortgood and bad fits. This procedure leads to an initial dataset, UP-3D, with richannotations. With a comprehensive set of experiments, we show how this data canbe used to train discriminative models that produce results with anunprecedented level of detail: our models predict 31 segments and 91 landmarklocations on the body. Using the 91 landmark pose estimator, we presentstate-of-the art results for 3D human pose and shape estimation using an orderof magnitude less training data and without assumptions about gender or pose inthe fitting procedure. We show that UP-3D can be enhanced with these improvedfits to grow in quantity and quality, which makes the system deployable onlarge scale. The data, code and models are available for research purposes.