Local feature frameworks are difficult to learn in an end-to-end fashion, dueto the discreteness inherent to the selection and matching of sparse keypoints.We introduce DISK (DIScrete Keypoints), a novel method that overcomes theseobstacles by leveraging principles from Reinforcement Learning (RL), optimizingend-to-end for a high number of correct feature matches. Our simple yetexpressive probabilistic model lets us keep the training and inference regimesclose, while maintaining good enough convergence properties to reliably trainfrom scratch. Our features can be extracted very densely while remainingdiscriminative, challenging commonly held assumptions about what constitutes agood keypoint, as showcased in Fig. 1, and deliver state-of-the-art results onthree public benchmarks.