BURST: A Benchmark for Unifying Object Recognition, Segmentation and Tracking in Video

Multiple existing benchmarks involve tracking and segmenting objects in videoe.g., Video Object Segmentation (VOS) and Multi-Object Tracking andSegmentation (MOTS), but there is little interaction between them due to theuse of disparate benchmark datasets and metrics (e.g. J&F, mAP, sMOTSA). As aresult, published works usually target a particular benchmark, and are noteasily comparable to each another. We believe that the development ofgeneralized methods that can tackle multiple tasks requires greater cohesionamong these research sub-communities. In this paper, we aim to facilitate thisby proposing BURST, a dataset which contains thousands of diverse videos withhigh-quality object masks, and an associated benchmark with six tasks involvingobject tracking and segmentation in video. All tasks are evaluated using thesame data and comparable metrics, which enables researchers to consider them inunison, and hence, more effectively pool knowledge from different methodsacross different tasks. Additionally, we demonstrate several baselines for alltasks and show that approaches for one task can be applied to another with aquantifiable and explainable performance difference. Dataset annotations andevaluation code is available at: https://github.com/Ali2500/BURST-benchmark.