CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation

This paper studies the complex task of simultaneous multi-object 3Dreconstruction, 6D pose and size estimation from a single-view RGB-Dobservation. In contrast to instance-level pose estimation, we focus on a morechallenging problem where CAD models are not available at inference time.Existing approaches mainly follow a complex multi-stage pipeline which firstlocalizes and detects each object instance in the image and then regresses toeither their 3D meshes or 6D poses. These approaches suffer fromhigh-computational cost and low performance in complex multi-object scenarios,where occlusions can be present. Hence, we present a simple one-stage approachto predict both the 3D shape and estimate the 6D pose and size jointly in abounding-box free manner. In particular, our method treats object instances asspatial centers where each center denotes the complete shape of an object alongwith its 6D pose and size. Through this per-pixel representation, our approachcan reconstruct in real-time (40 FPS) multiple novel object instances andpredict their 6D pose and sizes in a single-forward pass. Through extensiveexperiments, we demonstrate that our approach significantly outperforms allshape completion and categorical 6D pose and size estimation baselines onmulti-object ShapeNet and NOCS datasets respectively with a 12.6% absoluteimprovement in mAP for 6D pose for novel real-world object instances.