Learning to Refine Human Pose Estimation

Multi-person pose estimation in images and videos is an important yetchallenging task with many applications. Despite the large improvements inhuman pose estimation enabled by the development of convolutional neuralnetworks, there still exist a lot of difficult cases where even thestate-of-the-art models fail to correctly localize all body joints. Thismotivates the need for an additional refinement step that addresses thesechallenging cases and can be easily applied on top of any existing method. Inthis work, we introduce a pose refinement network (PoseRefiner) which takes asinput both the image and a given pose estimate and learns to directly predict arefined pose by jointly reasoning about the input-output space. In order forthe network to learn to refine incorrect body joint predictions, we employ anovel data augmentation scheme for training, where we model "hard" human posecases. We evaluate our approach on four popular large-scale pose estimationbenchmarks such as MPII Single- and Multi-Person Pose Estimation, PoseTrackPose Estimation, and PoseTrack Pose Tracking, and report systematic improvementover the state of the art.