MOTS: Multi-Object Tracking and Segmentation

This paper extends the popular task of multi-object tracking to multi-objecttracking and segmentation (MOTS). Towards this goal, we create densepixel-level annotations for two existing tracking datasets using asemi-automatic annotation procedure. Our new annotations comprise 65,213 pixelmasks for 977 distinct objects (cars and pedestrians) in 10,870 video frames.For evaluation, we extend existing multi-object tracking metrics to this newtask. Moreover, we propose a new baseline method which jointly addressesdetection, tracking, and segmentation with a single convolutional network. Wedemonstrate the value of our datasets by achieving improvements in performancewhen training on MOTS annotations. We believe that our datasets, metrics andbaseline will become a valuable resource towards developing multi-objecttracking approaches that go beyond 2D bounding boxes. We make our annotations,code, and models available at https://www.vision.rwth-aachen.de/page/mots.