Attributes act as intermediate representations that enable parameter sharingbetween classes, a must when training data is scarce. We propose to viewattribute-based image classification as a label-embedding problem: each classis embedded in the space of attribute vectors. We introduce a function thatmeasures the compatibility between an image and a label embedding. Theparameters of this function are learned on a training set of labeled samples toensure that, given an image, the correct classes rank higher than the incorrectones. Results on the Animals With Attributes and Caltech-UCSD-Birds datasetsshow that the proposed framework outperforms the standard Direct AttributePrediction baseline in a zero-shot learning scenario. Label embedding enjoys abuilt-in ability to leverage alternative sources of information instead of orin addition to attributes, such as e.g. class hierarchies or textualdescriptions. Moreover, label embedding encompasses the whole range of learningsettings from zero-shot learning to regular learning with a large number oflabeled examples.