Beyond Semantic to Instance Segmentation: Weakly-Supervised Instance Segmentation via Semantic Knowledge Transfer and Self-Refinement

Weakly-supervised instance segmentation (WSIS) has been considered as a morechallenging task than weakly-supervised semantic segmentation (WSSS). Comparedto WSSS, WSIS requires instance-wise localization, which is difficult toextract from image-level labels. To tackle the problem, most WSIS approachesuse off-the-shelf proposal techniques that require pre-training with instanceor object level labels, deviating the fundamental definition of thefully-image-level supervised setting. In this paper, we propose a novelapproach including two innovative components. First, we propose a semanticknowledge transfer to obtain pseudo instance labels by transferring theknowledge of WSSS to WSIS while eliminating the need for the off-the-shelfproposals. Second, we propose a self-refinement method to refine the pseudoinstance labels in a self-supervised scheme and to use the refined labels fortraining in an online manner. Here, we discover an erroneous phenomenon,semantic drift, that occurred by the missing instances in pseudo instancelabels categorized as background class. This semantic drift occurs confusionbetween background and instance in training and consequently degrades thesegmentation performance. We term this problem as semantic drift problem andshow that our proposed self-refinement method eliminates the semantic driftproblem. The extensive experiments on PASCAL VOC 2012 and MS COCO demonstratethe effectiveness of our approach, and we achieve a considerable performancewithout off-the-shelf proposal techniques. The code is available athttps://github.com/clovaai/BESTIE.