A Background-Agnostic Framework with Adversarial Training for Abnormal Event Detection in Video

Abnormal event detection in video is a complex computer vision problem thathas attracted significant attention in recent years. The complexity of the taskarises from the commonly-adopted definition of an abnormal event, that is, ararely occurring event that typically depends on the surrounding context.Following the standard formulation of abnormal event detection as outlierdetection, we propose a background-agnostic framework that learns from trainingvideos containing only normal events. Our framework is composed of an objectdetector, a set of appearance and motion auto-encoders, and a set ofclassifiers. Since our framework only looks at object detections, it can beapplied to different scenes, provided that normal events are definedidentically across scenes and that the single main factor of variation is thebackground. To overcome the lack of abnormal data during training, we proposean adversarial learning strategy for the auto-encoders. We create ascene-agnostic set of out-of-domain pseudo-abnormal examples, which arecorrectly reconstructed by the auto-encoders before applying gradient ascent onthe pseudo-abnormal examples. We further utilize the pseudo-abnormal examplesto serve as abnormal examples when training appearance-based and motion-basedbinary classifiers to discriminate between normal and abnormal latent featuresand reconstructions. We compare our framework with the state-of-the-art methodson four benchmark data sets, using various evaluation metrics. Compared toexisting methods, the empirical results indicate that our approach achievesfavorable performance on all data sets. In addition, we provide region-basedand track-based annotations for two large-scale abnormal event detection datasets from the literature, namely ShanghaiTech and Subway.