Large-scale, Fast and Accurate Shot Boundary Detection through Spatio-temporal Convolutional Neural Networks

Shot boundary detection (SBD) is an important pre-processing step for videomanipulation. Here, each segment of frames is classified as either sharp,gradual or no transition. Current SBD techniques analyze hand-crafted featuresand attempt to optimize both detection accuracy and processing speed. However,the heavy computations of optical flow prevents this. To achieve this aim, wepresent an SBD technique based on spatio-temporal Convolutional Neural Networks(CNN). Since current datasets are not large enough to train an accurate SBDCNN, we present a new dataset containing more than 3.5 million frames of sharpand gradual transitions. The transitions are generated synthetically usingimage compositing models. Our dataset contain additional 70,000 frames ofimportant hard-negative no transitions. We perform the largest evaluation todate for one SBD algorithm, on real and synthetic data, containing more than4.85 million frames. In comparison to the state of the art, we outperformdissolve gradual detection, generate competitive performance for sharpdetections and produce significant improvement in wipes. In addition, we are upto 11 times faster than the state of the art.