Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels

Training the multi-label image recognition models with partial labels, inwhich merely some labels are known while others are unknown for each image, isa considerably challenging and practical task. To address this task, currentalgorithms mainly depend on pre-training classification or similarity models togenerate pseudo labels for the unknown labels. However, these algorithms dependon sufficient multi-label annotations to train the models, leading to poorperformance especially with low known label proportion. In this work, wepropose to blend category-specific representation across different images totransfer information of known labels to complement unknown labels, which canget rid of pre-training models and thus does not depend on sufficientannotations. To this end, we design a unified semantic-aware representationblending (SARB) framework that exploits instance-level and prototype-levelsemantic representation to complement unknown labels by two complementarymodules: 1) an instance-level representation blending (ILRB) module blends therepresentations of the known labels in an image to the representations of theunknown labels in another image to complement these unknown labels. 2) aprototype-level representation blending (PLRB) module learns more stablerepresentation prototypes for each category and blends the representation ofunknown labels with the prototypes of corresponding labels to complement theselabels. Extensive experiments on the MS-COCO, Visual Genome, Pascal VOC 2007datasets show that the proposed SARB framework obtains superior performanceover current leading competitors on all known label proportion settings, i.e.,with the mAP improvement of 4.6%, 4.%, 2.2% on these three datasets when theknown label proportion is 10%. Codes are available athttps://github.com/HCPLab-SYSU/HCP-MLR-PL.