One-sentence Summary

Meta AI Research et al. propose MAUBERT, a multilingual extension of HuBERT that maps phones to articulatory features across 55 languages to learn language-independent phonetic representations, demonstrating superior context-invariant ABX discriminability over self-supervised baselines and effective few-shot adaptation to unseen languages with only 10 hours of fine-tuning.

Key Contributions

• MAUBERT extends HuBERT by continuing pre-training on 55 languages with supervision from a phonetic-to-articulatory feature mapping, enabling the model to predict articulatory features or phones. This training produces language-independent representations that capture multilingual phonetic properties.

• ABX discriminability tests show that MAUBERT yields more context-invariant representations than state-of-the-art multilingual self-supervised models. The evaluation also compares read and casual speech, demonstrating adaptation to casual speech with minimal self-supervised fine-tuning of 10 hours.

• The models adapt effectively to unseen and low-resource languages, including Wolof with less than 2 hours of data. These results show that supervised articulatory feature prediction can instill linguistic inductive biases in self-supervised speech models.

Introduction

Automatic discovery of linguistic units from raw audio matters for speech technology in low resource languages and for modeling early language acquisition. Self-supervised speech models have improved phonetic discriminability compared with conventional features, but their discovered units usually are shorter and more numerous than phones, remain sensitive to speaker and context, and often require massive clean training data. The authors propose MAUBERT, which fine-tunes HuBERT on IPA phones and articulatory features across 55 languages to inject explicit phonological inductive bias. This produces robust multilingual phonetic representations that adapt to new languages with only about 10 hours of unlabeled speech and can suggest candidate phoneme and feature inventories.

Dataset

• The authors base the dataset on the VoxCommunis Corpus, which provides phone-level annotations for a subset of Common Voice produced with the Montreal Forced Aligner.

• Across 63 VoxCommunis languages, 55 are used for supervised articulatory feature prediction, totaling 788.4 hours; 5 languages serve as development languages; and 3 are excluded from supervised training because they are used as test languages.

• Phone-level annotations are converted into ternary articulatory feature labels using PanPhon’s feature table. Segments incompatible with PanPhon are manually corrected, and the IPA segment inventory is collapsed to distinct feature vectors, reducing it from 6367 to 3293 representative segments.

• These collapsed representatives are used as targets for both phone recognition and underlying feature-value recognition.

• For few-shot language evaluation, the authors curate ABX-ready datasets for five development languages: Swahili, Tamil, Thai, Turkish, and Ukrainian. Each language has three splits: a 10-hour training set, a validation set, and a test set.

• For Zero Resource Speech Challenge 2017 test languages, namely English, French, Mandarin, German, and Wolof, the authors extract training and validation splits of up to 10 hours from the original corpora. The original test subsets are kept at their existing lengths of 1 second, 10 seconds, and 120 seconds to evaluate context-length effects.

• An additional ABX-ready dataset is built from high-quality casual speech conversations in English and French, with the same three-split structure as the development language datasets.

Method

The authors propose MAUBERT, a multilingual speech model for recognizing articulatory features and phones from raw audio. The method combines continual pretraining of a self-supervised speech encoder with supervised feature or phone recognition, followed by self-supervised adaptation to new languages through pseudo-labeling and masked language modeling. Refer to the framework diagram.

The encoder backbone is initialized from HuBERT-base. The convolutional feature extractor is kept frozen, while the Transformer encoder remains trainable during multilingual continual pretraining. Input masking is disabled in this stage, following the design used for downstream speech processing. The model receives the output of the feature extractor after layer normalization and dropout, as well as the hidden states produced by each of the twelve Transformer encoder layers. These multiple levels of representation are intended to provide complementary information, since different Transformer depths may encode different acoustic and linguistic regularities.

On top of the encoder, the authors use a downstream module rather than a single linear classification head. The intermediate encoder representations are first combined through a learnable weighted sum. This weighted representation is then projected upward into a 1024-dimensional space and passed through a bidirectional two-layer LSTM. The bidirectional LSTM gives the system access to contextual information from both left and right directions. The forward and backward LSTM states are concatenated and projected down into the target output space. For articulatory feature recognition, this output space has dimensionality $22$22, while for direct phone recognition it has dimensionality $3293$3293.

Two model variants are built from this architecture. MAUBERT-FEAT uses an articulatory feature bottleneck, where the network first predicts a compact representation of phonetic features. MAUBERT-PHONE bypasses the intermediate feature layer and predicts phones directly. The feature-based variant is motivated by the hypothesis that a linguistically structured representation may transfer better across languages. Since the mapping from articulatory features to phones is not injective, MAUBERT-FEAT also includes a phone model. This phone model is implemented as a two-layer perceptron and is trained jointly with the feature predictor. To ensure that the main representation is shaped primarily by the articulatory feature objective, a stop-gradient operator prevents the phone recognition loss from backpropagating into the feature hidden states.

Training is performed at the frame level. For MAUBERT-FEAT, the feature recognition objective uses binary cross-entropy because the articulatory features are represented with ternary values. For phone recognition, the objective uses multiclass cross-entropy over the phone inventory. The model is optimized with Adam and a staged learning rate schedule. Because the training data are multilingual, the authors also apply language upsampling to balance the contribution of low-resource and high-resource languages during supervised pretraining.

The self-supervised adaptation stage is designed for languages with limited or no labeled data. This stage uses a teacher-student setup. The teacher network is the pretrained MAUBERT model, which produces representations for the target-language speech. These representations are clustered offline to create pseudo-labels. A small number of target-language frames are masked, and the student network, based on the HuBERT encoder, is trained to predict the cluster identity of the masked positions. This masked prediction objective allows the model to adapt to new acoustic patterns without requiring transcriptions, phone annotations, or feature annotations.

The pseudo-label generation process can use different sources. One option is K-means clustering, applied to representations extracted from either the Transformer encoder or the downstream module. The clustering uses $K = 100$K=100 clusters, producing target cluster labels of the form $\{c_k\}_{k=1}^{K}${ck​}k=1K​. For MAUBERT-FEAT, an alternative is to use frequent articulatory feature vectors by selecting the most common feature representations produced by the model. For both MAUBERT variants, the pseudo-labels can instead come from frequent phones predicted on the target data or from the phone inventory derived during multilingual pretraining. These variants let the adaptation process choose between continuous clustering of learned representations and supervision from discrete phonetic or articulatory units.

Experiment

The experiments evaluate MAUBERT models using multilingual phone recognition metrics and ABX phonetic-invariance tests, comparing feature-based and phone-based variants across training, development, and zero-shot or few-shot language-adaptation settings. Results show that the feature-based model better predicts articulatory features, while the phone-based model gives stronger phone recognition and zero-shot phonetic invariance. Supervised fine-tuning, especially masked phone prediction, provides the largest gains, and self-supervised fine-tuning with phone-frequency clustering also improves robustness across languages and speaking styles, while frequency-threshold methods help balance precision and recall in phonetic inventory discovery.

MAUBERT performs better on training languages than on development languages, with phone recognition metrics showing a larger decline than feature accuracy. The feature-based variant is stronger for articulatory feature prediction, but the phone-based variant achieves slightly better phone recognition performance. Phone accuracy and phone error rate degrade more than feature accuracy when moving to unseen development languages. The phone-based variant has a small advantage in phone accuracy and a larger advantage in phone error rate.

MAUBERT sits between compact monolingual speech models and very large multilingual baselines, using a moderate model size and far fewer training hours than the mass-multilingual systems. It also avoids exposing development languages during training, making it suitable for zero-shot adaptation to new languages. This comparison supports the finding that effective language adaptation can rely on targeted model design rather than extreme multilingual scale. MAUBERT provides broad language coverage with a moderate number of parameters compared with much larger multilingual speech models. It uses far less training data than MMS and XEUS while still targeting multiple evaluation languages. Unlike the multilingual baselines, MAUBERT does not see development languages during training, strengthening the zero-shot claim. Its compact multilingual scale helps keep a fair comparison with monolingual baselines while still offering multilingual capability.

Lower ABX scores indicate stronger acoustic discriminability, and the zero-shot comparison shows that multilingual self-supervised models are much more effective than MFCC baselines. The development languages tend to produce higher error scores than the test languages under comparable conditions, implying that some language families are more difficult for crosslingual phonetic discrimination. Supervised fine-tuning further lowers ABX error rates, with masked-input phone prediction providing especially large gains for triphone discrimination. XEUS achieves better zero-shot average ABX scores than MMS-1B and mHuBERT-147 across development and test languages. Development-language ABX scores are generally higher than test-language scores, indicating harder phonetic discrimination tasks. Supervised fine-tuning reduces ABX error rates relative to zero-shot settings, and masked-input prediction is reported as especially effective.

Triphone ABX error rates across English and French registers show a clear gap between read and spontaneous speech, with spontaneous speech being more difficult. Among the listed zero-shot systems, XEUS has the lowest error rates overall, while MAUBERT-FEAT is competitive, especially in read speech. Self-supervised fine-tuning is noted to improve adaptation and help recover performance on spontaneous speech. Spontaneous speech leads to higher triphone ABX error rates than read speech. XEUS is strongest among baselines, and MAUBERT-FEAT remains competitive in read conditions.

The experiments evaluate MAUBERT across training and unseen development languages, showing that phone recognition degrades more than articulatory feature prediction, with phone-based and feature-based variants excelling at phone recognition and feature prediction respectively. The models are compared against monolingual and large multilingual baselines under a compact, zero-shot setting, supporting the view that efficient model design rather than extreme multilingual scale can enable effective cross-language adaptation. ABX results further indicate that multilingual self-supervised models greatly outperform MFCC baselines, that development languages are generally harder than test languages, and that supervised fine-tuning, especially masked-input phone prediction, improves discriminability. Additional analysis of English and French registers finds spontaneous speech more difficult than read speech, while MAUBERT-FEAT remains competitive overall.