Efficient Training of Audio Transformers with Patchout

The great success of transformer-based models in natural language processing(NLP) has led to various attempts at adapting these architectures to otherdomains such as vision and audio. Recent work has shown that transformers canoutperform Convolutional Neural Networks (CNNs) on vision and audio tasks.However, one of the main shortcomings of transformer models, compared to thewell-established CNNs, is the computational complexity. In transformers, thecompute and memory complexity is known to grow quadratically with the inputlength. Therefore, there has been extensive work on optimizing transformers,but often at the cost of degrading predictive performance. In this work, wepropose a novel method to optimize and regularize transformers on audiospectrograms. Our proposed models achieve a new state-of-the-art performance onAudioset and can be trained on a single consumer-grade GPU. Furthermore, wepropose a transformer model that outperforms CNNs in terms of both performanceand training speed. Source code: https://github.com/kkoutini/PaSST