Interpretable and Generalizable Person Re-Identification with Query-Adaptive Convolution and Temporal Lifting

For person re-identification, existing deep networks often focus onrepresentation learning. However, without transfer learning, the learned modelis fixed as is, which is not adaptable for handling various unseen scenarios.In this paper, beyond representation learning, we consider how to formulateperson image matching directly in deep feature maps. We treat image matching asfinding local correspondences in feature maps, and construct query-adaptiveconvolution kernels on the fly to achieve local matching. In this way, thematching process and results are interpretable, and this explicit matching ismore generalizable than representation features to unseen scenarios, such asunknown misalignments, pose or viewpoint changes. To facilitate end-to-endtraining of this architecture, we further build a class memory module to cachefeature maps of the most recent samples of each class, so as to compute imagematching losses for metric learning. Through direct cross-dataset evaluation,the proposed Query-Adaptive Convolution (QAConv) method gains largeimprovements over popular learning methods (about 10%+ mAP), and achievescomparable results to many transfer learning methods. Besides, a model-freetemporal cooccurrence based score weighting method called TLift is proposed,which improves the performance to a further extent, achieving state-of-the-artresults in cross-dataset person re-identification. Code is available athttps://github.com/ShengcaiLiao/QAConv.