Research in auditory, visual, and audiovisual speech recognition (ASR, VSR,and AVSR, respectively) has traditionally been conducted independently. Evenrecent self-supervised studies addressing two or all three tasks simultaneouslytend to yield separate models, leading to disjoint inference pipelines withincreased memory requirements and redundancies. This paper proposes unifiedtraining strategies for these systems. We demonstrate that training a singlemodel for all three tasks enhances VSR and AVSR performance, overcoming typicaloptimisation challenges when training from scratch. Moreover, we introduce agreedy pseudo-labelling approach to more effectively leverage unlabelledsamples, addressing shortcomings in related self-supervised methods. Finally,we develop a self-supervised pre-training method within our framework, provingits effectiveness alongside our semi-supervised approach. Despite using asingle model for all tasks, our unified approach achieves state-of-the-artperformance compared to recent methods on LRS3 and LRS2 for ASR, VSR, and AVSR,as well as on the newly released WildVSR dataset. Code and models are availableat https://github.com/ahaliassos/usr.