BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models

Tracking the 6D pose of objects in video sequences is important for robotmanipulation. Most prior efforts, however, often assume that the targetobject's CAD model, at least at a category-level, is available for offlinetraining or during online template matching. This work proposes BundleTrack, ageneral framework for 6D pose tracking of novel objects, which does not dependupon 3D models, either at the instance or category-level. It leverages thecomplementary attributes of recent advances in deep learning for segmentationand robust feature extraction, as well as memory-augmented pose graphoptimization for spatiotemporal consistency. This enables long-term, low-drifttracking under various challenging scenarios, including significant occlusionsand object motions. Comprehensive experiments given two public benchmarksdemonstrate that the proposed approach significantly outperforms state-of-art,category-level 6D tracking or dynamic SLAM methods. When compared againststate-of-art methods that rely on an object instance CAD model, comparableperformance is achieved, despite the proposed method's reduced informationrequirements. An efficient implementation in CUDA provides a real-timeperformance of 10Hz for the entire framework. Code is available at:https://github.com/wenbowen123/BundleTrack