COTR: Correspondence Transformer for Matching Across Images

We propose a novel framework for finding correspondences in images based on adeep neural network that, given two images and a query point in one of them,finds its correspondence in the other. By doing so, one has the option to queryonly the points of interest and retrieve sparse correspondences, or to queryall points in an image and obtain dense mappings. Importantly, in order tocapture both local and global priors, and to let our model relate between imageregions using the most relevant among said priors, we realize our network usinga transformer. At inference time, we apply our correspondence network byrecursively zooming in around the estimates, yielding a multiscale pipelineable to provide highly-accurate correspondences. Our method significantlyoutperforms the state of the art on both sparse and dense correspondenceproblems on multiple datasets and tasks, ranging from wide-baseline stereo tooptical flow, without any retraining for a specific dataset. We commit toreleasing data, code, and all the tools necessary to train from scratch andensure reproducibility.