Jointformer: Single-Frame Lifting Transformer with Error Prediction and Refinement for 3D Human Pose Estimation

Monocular 3D human pose estimation technologies have the potential to greatlyincrease the availability of human movement data. The best-performing modelsfor single-image 2D-3D lifting use graph convolutional networks (GCNs) thattypically require some manual input to define the relationships betweendifferent body joints. We propose a novel transformer-based approach that usesthe more generalised self-attention mechanism to learn these relationshipswithin a sequence of tokens representing joints. We find that the use ofintermediate supervision, as well as residual connections between the stackedencoders benefits performance. We also suggest that using error prediction aspart of a multi-task learning framework improves performance by allowing thenetwork to compensate for its confidence level. We perform extensive ablationstudies to show that each of our contributions increases performance.Furthermore, we show that our approach outperforms the recent state of the artfor single-frame 3D human pose estimation by a large margin. Our code andtrained models are made publicly available on Github.