Mitigating Background Shift in Class-Incremental Semantic Segmentation

Class-Incremental Semantic Segmentation(CISS) aims to learn new classeswithout forgetting the old ones, using only the labels of the new classes. Toachieve this, two popular strategies are employed: 1) pseudo-labeling andknowledge distillation to preserve prior knowledge; and 2) background weighttransfer, which leverages the broad coverage of background in learning newclasses by transferring background weight to the new class classifier. However,the first strategy heavily relies on the old model in detecting old classeswhile undetected pixels are regarded as the background, thereby leading to thebackground shift towards the old classes(i.e., misclassification of old classas background). Additionally, in the case of the second approach, initializingthe new class classifier with background knowledge triggers a similarbackground shift issue, but towards the new classes. To address these issues,we propose a background-class separation framework for CISS. To begin with,selective pseudo-labeling and adaptive feature distillation are to distill onlytrustworthy past knowledge. On the other hand, we encourage the separationbetween the background and new classes with a novel orthogonal objective alongwith label-guided output distillation. Our state-of-the-art results validatethe effectiveness of these proposed methods.