Detect Everything with Few Examples

Few-shot object detection aims at detecting novel categories given only a fewexample images. It is a basic skill for a robot to perform tasks in openenvironments. Recent methods focus on finetuning strategies, with complicatedprocedures that prohibit a wider application. In this paper, we introduceDE-ViT, a few-shot object detector without the need for finetuning. DE-ViT'snovel architecture is based on a new region-propagation mechanism forlocalization. The propagated region masks are transformed into bounding boxesthrough a learnable spatial integral layer. Instead of training prototypeclassifiers, we propose to use prototypes to project ViT features into asubspace that is robust to overfitting on base classes. We evaluate DE-ViT onfew-shot, and one-shot object detection benchmarks with Pascal VOC, COCO, andLVIS. DE-ViT establishes new state-of-the-art results on all benchmarks.Notably, for COCO, DE-ViT surpasses the few-shot SoTA by 15 mAP on 10-shot and7.2 mAP on 30-shot and one-shot SoTA by 2.8 AP50. For LVIS, DE-ViT outperformsfew-shot SoTA by 17 box APr. Further, we evaluate DE-ViT with a real robot bybuilding a pick-and-place system for sorting novel objects based on exampleimages. The videos of our robot demonstrations, the source code and the modelsof DE-ViT can be found at https://mlzxy.github.io/devit.