Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval

Optimising a ranking-based metric, such as Average Precision (AP), isnotoriously challenging due to the fact that it is non-differentiable, andhence cannot be optimised directly using gradient-descent methods. To this end,we introduce an objective that optimises instead a smoothed approximation ofAP, coined Smooth-AP. Smooth-AP is a plug-and-play objective function thatallows for end-to-end training of deep networks with a simple and elegantimplementation. We also present an analysis for why directly optimising theranking based metric of AP offers benefits over other deep metric learninglosses. We apply Smooth-AP to standard retrieval benchmarks: Stanford Onlineproducts and VehicleID, and also evaluate on larger-scale datasets: INaturalistfor fine-grained category retrieval, and VGGFace2 and IJB-C for face retrieval.In all cases, we improve the performance over the state-of-the-art, especiallyfor larger-scale datasets, thus demonstrating the effectiveness and scalabilityof Smooth-AP to real-world scenarios.