Robust Self-Supervised Audio-Visual Speech Recognition

Audio-based automatic speech recognition (ASR) degrades significantly innoisy environments and is particularly vulnerable to interfering speech, as themodel cannot determine which speaker to transcribe. Audio-visual speechrecognition (AVSR) systems improve robustness by complementing the audio streamwith the visual information that is invariant to noise and helps the modelfocus on the desired speaker. However, previous AVSR work focused solely on thesupervised learning setup; hence the progress was hindered by the amount oflabeled data available. In this work, we present a self-supervised AVSRframework built upon Audio-Visual HuBERT (AV-HuBERT), a state-of-the-artaudio-visual speech representation learning model. On the largest availableAVSR benchmark dataset LRS3, our approach outperforms prior state-of-the-art by~50% (28.0% vs. 14.1%) using less than 10% of labeled data (433hr vs. 30hr) inthe presence of babble noise, while reducing the WER of an audio-based model byover 75% (25.8% vs. 5.8%) on average.