Attribute Prototype Network for Any-Shot Learning

Any-shot image classification allows to recognize novel classes with only afew or even zero samples. For the task of zero-shot learning, visual attributeshave been shown to play an important role, while in the few-shot regime, theeffect of attributes is under-explored. To better transfer attribute-basedknowledge from seen to unseen classes, we argue that an image representationwith integrated attribute localization ability would be beneficial forany-shot, i.e. zero-shot and few-shot, image classification tasks. To this end,we propose a novel representation learning framework that jointly learnsdiscriminative global and local features using only class-level attributes.While a visual-semantic embedding layer learns global features, local featuresare learned through an attribute prototype network that simultaneouslyregresses and decorrelates attributes from intermediate features. Furthermore,we introduce a zoom-in module that localizes and crops the informative regionsto encourage the network to learn informative features explicitly. We show thatour locality augmented image representations achieve a new state-of-the-art onchallenging benchmarks, i.e. CUB, AWA2, and SUN. As an additional benefit, ourmodel points to the visual evidence of the attributes in an image, confirmingthe improved attribute localization ability of our image representation. Theattribute localization is evaluated quantitatively with ground truth partannotations, qualitatively with visualizations, and through well-designed userstudies.