HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions

Reconstructing 3D hand mesh robustly from a single image is very challenging,due to the lack of diversity in existing real-world datasets. While datasynthesis helps relieve the issue, the syn-to-real gap still hinders its usage.In this work, we present HandBooster, a new approach to uplift the datadiversity and boost the 3D hand-mesh reconstruction performance by training aconditional generative space on hand-object interactions and purposely samplingthe space to synthesize effective data samples. First, we construct versatilecontent-aware conditions to guide a diffusion model to produce realistic imageswith diverse hand appearances, poses, views, and backgrounds; favorably,accurate 3D annotations are obtained for free. Then, we design a novelcondition creator based on our similarity-aware distribution samplingstrategies to deliberately find novel and realistic interaction poses that aredistinctive from the training set. Equipped with our method, several baselinescan be significantly improved beyond the SOTA on the HO3D and DexYCBbenchmarks. Our code will be released onhttps://github.com/hxwork/HandBooster_Pytorch.