Multi-label image recognition with partial labels (MLR-PL), in which somelabels are known while others are unknown for each image, may greatly reducethe cost of annotation and thus facilitate large-scale MLR. We find that strongsemantic correlations exist within each image and across different images, andthese correlations can help transfer the knowledge possessed by the knownlabels to retrieve the unknown labels and thus improve the performance of theMLR-PL task (see Figure 1). In this work, we propose a novel heterogeneoussemantic transfer (HST) framework that consists of two complementary transfermodules that explore both within-image and cross-image semantic correlations totransfer the knowledge possessed by known labels to generate pseudo labels forthe unknown labels. Specifically, an intra-image semantic transfer (IST) modulelearns an image-specific label co-occurrence matrix for each image and maps theknown labels to complement the unknown labels based on these matrices.Additionally, a cross-image transfer (CST) module learns category-specificfeature-prototype similarities and then helps complement the unknown labelsthat have high degrees of similarity with the corresponding prototypes.Finally, both the known and generated pseudo labels are used to train MLRmodels. Extensive experiments conducted on the Microsoft COCO, Visual Genome,and Pascal VOC 2007 datasets show that the proposed HST framework achievessuperior performance to that of current state-of-the-art algorithms.Specifically, it obtains mean average precision (mAP) improvements of 1.4%,3.3%, and 0.4% on the three datasets over the results of the best-performingpreviously developed algorithm.