3D Instance Segmentation via Multi-Task Metric Learning

We propose a novel method for instance label segmentation of dense 3D voxelgrids. We target volumetric scene representations, which have been acquiredwith depth sensors or multi-view stereo methods and which have been processedwith semantic 3D reconstruction or scene completion methods. The main task isto learn shape information about individual object instances in order toaccurately separate them, including connected and incompletely scanned objects.We solve the 3D instance-labeling problem with a multi-task learning strategy.The first goal is to learn an abstract feature embedding, which groups voxelswith the same instance label close to each other while separating clusters withdifferent instance labels from each other. The second goal is to learn instanceinformation by densely estimating directional information of the instance'scenter of mass for each voxel. This is particularly useful to find instanceboundaries in the clustering post-processing step, as well as, for scoring thesegmentation quality for the first goal. Both synthetic and real-worldexperiments demonstrate the viability and merits of our approach. In fact, itachieves state-of-the-art performance on the ScanNet 3D instance segmentationbenchmark.