SynthDistill: Face Recognition with Knowledge Distillation from Synthetic Data

State-of-the-art face recognition networks are often computationallyexpensive and cannot be used for mobile applications. Training lightweight facerecognition models also requires large identity-labeled datasets. Meanwhile,there are privacy and ethical concerns with collecting and using large facerecognition datasets. While generating synthetic datasets for training facerecognition models is an alternative option, it is challenging to generatesynthetic data with sufficient intra-class variations. In addition, there isstill a considerable gap between the performance of models trained on real andsynthetic data. In this paper, we propose a new framework (named SynthDistill)to train lightweight face recognition models by distilling the knowledge of apretrained teacher face recognition model using synthetic data. We use apretrained face generator network to generate synthetic face images and use thesynthesized images to learn a lightweight student network. We use syntheticface images without identity labels, mitigating the problems in the intra-classvariation generation of synthetic datasets. Instead, we propose a novel dynamicsampling strategy from the intermediate latent space of the face generatornetwork to include new variations of the challenging images while furtherexploring new face images in the training batch. The results on five differentface recognition datasets demonstrate the superiority of our lightweight modelcompared to models trained on previous synthetic datasets, achieving averification accuracy of 99.52% on the LFW dataset with a lightweight network.The results also show that our proposed framework significantly reduces the gapbetween training with real and synthetic data. The source code for replicatingthe experiments is publicly released.