Exploring Dual Model Knowledge Distillation for Anomaly Detection

Unsupervised anomaly detection holds significant importance in large-scale industrial manufacturing. Recent methods have capitalized on the benefits of utilizinga classifier pretrained on natural images to extract representative features fromspecific layers. These extracted features are subsequently processed using varioustechniques. Notably, memory bank-based methods have demonstrated exceptional accuracy; however, they often incur a trade-off in terms of latency.This latency trade-off poses a challenge in real-time industrial applicationswhere prompt anomaly detection and response are crucial. Indeed, alternativeapproaches such as knowledge distillation and normalized flow have demonstratedpromising performance in unsupervised anomaly detection while maintaining lowlatency. In this paper, we aim to revisit the concept of knowledge distillationin the context of unsupervised anomaly detection, emphasizing the significanceof feature selection. By employing distinctive features and leveraging differentmodels, we intend to highlight the importance of carefully selecting and utilizing relevant features specifically tailored for the task of anomaly detection. Thisarticle introduces a novel approach based on dual model knowledge distillationfor anomaly detection. The proposed method leverages both deep and shallowlayers to incorporate various types of semantic information.