We show that existing upsampling operators can be unified using the notion ofthe index function. This notion is inspired by an observation in the decodingprocess of deep image matting where indices-guided unpooling can often recoverboundary details considerably better than other upsampling operators such asbilinear interpolation. By viewing the indices as a function of the featuremap, we introduce the concept of "learning to index", and present a novelindex-guided encoder-decoder framework where indices are self-learnedadaptively from data and are used to guide the downsampling and upsamplingstages, without extra training supervision. At the core of this framework is anew learnable module, termed Index Network (IndexNet), which dynamicallygenerates indices conditioned on the feature map itself. IndexNet can be usedas a plug-in applying to almost all off-the-shelf convolutional networks thathave coupled downsampling and upsampling stages, giving the networks theability to dynamically capture variations of local patterns. In particular, weinstantiate and investigate five families of IndexNet and demonstrate theireffectiveness on four dense prediction tasks, including image denoising, imagematting, semantic segmentation, and monocular depth estimation. Code and modelshave been made available at: https://tinyurl.com/IndexNetV1