Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses

We present Neural-Guided RANSAC (NG-RANSAC), an extension to the classicRANSAC algorithm from robust optimization. NG-RANSAC uses prior information toimprove model hypothesis search, increasing the chance of finding outlier-freeminimal sets. Previous works use heuristic side-information like hand-crafteddescriptor distance to guide hypothesis search. In contrast, we learnhypothesis search in a principled fashion that lets us optimize an arbitrarytask loss during training, leading to large improvements on classic computervision tasks. We present two further extensions to NG-RANSAC. Firstly, usingthe inlier count itself as training signal allows us to train neural guidancein a self-supervised fashion. Secondly, we combine neural guidance withdifferentiable RANSAC to build neural networks which focus on certain parts ofthe input data and make the output predictions as good as possible. We evaluateNG-RANSAC on a wide array of computer vision tasks, namely estimation ofepipolar geometry, horizon line estimation and camera re-localization. Weachieve superior or competitive results compared to state-of-the-art robustestimators, including very recent, learned ones.