Joint Reconstruction of 3D Human and Object via Contact-Based Refinement Transformer

Human-object contact serves as a strong cue to understand how humansphysically interact with objects. Nevertheless, it is not widely explored toutilize human-object contact information for the joint reconstruction of 3Dhuman and object from a single image. In this work, we present a novel joint 3Dhuman-object reconstruction method (CONTHO) that effectively exploits contactinformation between humans and objects. There are two core designs in oursystem: 1) 3D-guided contact estimation and 2) contact-based 3D human andobject refinement. First, for accurate human-object contact estimation, CONTHOinitially reconstructs 3D humans and objects and utilizes them as explicit 3Dguidance for contact estimation. Second, to refine the initial reconstructionsof 3D human and object, we propose a novel contact-based refinement Transformerthat effectively aggregates human features and object features based on theestimated human-object contact. The proposed contact-based refinement preventsthe learning of erroneous correlation between human and object, which enablesaccurate 3D reconstruction. As a result, our CONTHO achieves state-of-the-artperformance in both human-object contact estimation and joint reconstruction of3D human and object. The code is publicly available athttps://github.com/dqj5182/CONTHO_RELEASE.