Discriminative Region-based Multi-Label Zero-Shot Learning

Multi-label zero-shot learning (ZSL) is a more realistic counter-part ofstandard single-label ZSL since several objects can co-exist in a naturalimage. However, the occurrence of multiple objects complicates the reasoningand requires region-specific processing of visual features to preserve theircontextual cues. We note that the best existing multi-label ZSL method takes ashared approach towards attending to region features with a common set ofattention maps for all the classes. Such shared maps lead to diffusedattention, which does not discriminatively focus on relevant locations when thenumber of classes are large. Moreover, mapping spatially-pooled visual featuresto the class semantics leads to inter-class feature entanglement, thushampering the classification. Here, we propose an alternate approach towardsregion-based discriminability-preserving multi-label zero-shot classification.Our approach maintains the spatial resolution to preserve region-levelcharacteristics and utilizes a bi-level attention module (BiAM) to enrich thefeatures by incorporating both region and scene context information. Theenriched region-level features are then mapped to the class semantics and onlytheir class predictions are spatially pooled to obtain image-level predictions,thereby keeping the multi-class features disentangled. Our approach sets a newstate of the art on two large-scale multi-label zero-shot benchmarks: NUS-WIDEand Open Images. On NUS-WIDE, our approach achieves an absolute gain of 6.9%mAP for ZSL, compared to the best published results.