Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild

Recognizing scenes and objects in 3D from a single image is a longstandinggoal of computer vision with applications in robotics and AR/VR. For 2Drecognition, large datasets and scalable solutions have led to unprecedentedadvances. In 3D, existing benchmarks are small in size and approachesspecialize in few object categories and specific domains, e.g. urban drivingscenes. Motivated by the success of 2D recognition, we revisit the task of 3Dobject detection by introducing a large benchmark, called Omni3D. Omni3Dre-purposes and combines existing datasets resulting in 234k images annotatedwith more than 3 million instances and 98 categories. 3D detection at suchscale is challenging due to variations in camera intrinsics and the richdiversity of scene and object types. We propose a model, called Cube R-CNN,designed to generalize across camera and scene types with a unified approach.We show that Cube R-CNN outperforms prior works on the larger Omni3D andexisting benchmarks. Finally, we prove that Omni3D is a powerful dataset for 3Dobject recognition and show that it improves single-dataset performance and canaccelerate learning on new smaller datasets via pre-training.