We introduce point affiliation into feature upsampling, a notion thatdescribes the affiliation of each upsampled point to a semantic cluster formedby local decoder feature points with semantic similarity. By rethinking pointaffiliation, we present a generic formulation for generating upsamplingkernels. The kernels encourage not only semantic smoothness but also boundarysharpness in the upsampled feature maps. Such properties are particularlyuseful for some dense prediction tasks such as semantic segmentation. The keyidea of our formulation is to generate similarity-aware kernels by comparingthe similarity between each encoder feature point and the spatially associatedlocal region of decoder features. In this way, the encoder feature point canfunction as a cue to inform the semantic cluster of upsampled feature points.To embody the formulation, we further instantiate a lightweight upsamplingoperator, termed Similarity-Aware Point Affiliation (SAPA), and investigate itsvariants. SAPA invites consistent performance improvements on a number of denseprediction tasks, including semantic segmentation, object detection, depthestimation, and image matting. Code is available at:https://github.com/poppinace/sapa