One-sentence Summary

Researchers at the University of Notre Dame fine-tuned Wav2Vec2-XLSR-53 on 25 languages spanning 15 writing systems and found that high logographicity, a measure of orthographic complexity, correlates with degraded ASR accuracy, while phonological complexity shows no strong correlation.

Key Contributions

• The study fine-tunes Wav2Vec2-XLSR-53 on 25 languages and multiple writing systems, comparing ASR accuracy with grapheme counts, unigram grapheme entropy, logographicity, and phoneme counts.
• The study separates orthographic complexity from phonological complexity by measuring logographicity with phoneme-to-grapheme model attention dispersion and comparing it with phonological inventory size.
• The results show character error rate strongly correlates with orthographic complexity, especially logographicity and grapheme inventory size, while phonological inventory size shows no significant correlation; complex orthographies also produce slower, less stable, and less accurate learning.

Introduction

Automatic speech recognition must work across languages whose writing and sound systems differ in difficulty, which matters for building inclusive and low-resource speech technologies. Prior multilingual ASR work has improved performance through self-supervised pretraining, yet it has not clearly separated how orthographic complexity, such as large grapheme inventories and logographicity, affects fine-tuned accuracy, especially when pretraining data can bias evaluation. The authors leverage Wav2Vec2-XLSR-53 fine-tuning across 25 languages and 15 orthographies to test whether grapheme inventory size, logographicity, and phoneme inventory size reduce ASR accuracy. Their results show that orthographic complexity strongly correlates with higher character error rates, while phonological inventory size does not, suggesting these models adapt more readily to sound systems than to complex writing systems.

Dataset

• The speech training corpus is built primarily from Common Voice 16.1 for all examined languages except English and Korean. The authors replace English with LibriSpeech because Common Voice English includes many non-native speech samples, and they replace Korean with Zeroth-Korean because the Korean subset in Common Voice 16.1 lacks enough samples.

• Long audio samples are filtered out for LibriSpeech and Zeroth-Korean because these sources allow longer maximum audio lengths than Common Voice 16.1. For each experimental setting, the authors continue extracting training samples until the total audio duration reaches 10,000 seconds, standardizing the dataset size by speech length rather than by number of samples.

• For the logographicity analysis, the authors use language-specific datasets listed in Table 1, covering languages with different writing systems. For each language, they extract 10,000 samples and split them into an 80% training set and a 20% validation set. No test set is created because the analysis measures attention spread rather than transcription accuracy.

• Text samples are preprocessed by removing punctuation and omitting samples containing numerical symbols. The phoneme sequence schema contains seven tokens: three context tokens to the left of the target word, the target token in the middle, and three context tokens to the right. The target word is enclosed by <TARG> and </TARG>, word boundaries are marked by [SEP], and [START] and [END] are added to the beginning and end of both the input and output sequences.

• Phonemization is handled by language-specific tools. The same converters are used for Japanese, Korean, and Chinese. Thai is romanized using PyThaiNLP’s transliteration module. English is phonemized with the CMU Pronouncing Dictionary via the eng-to-ipa package. Other alphabetic languages and Arabic are converted to IPA using the gruut library.

• The speech data are used to create length-controlled training sets for the language-specific speech experiments. The cleaned and phonemized text data are used to train phoneme-to-orthography models and measure whether the model attends to local pronunciation alone or to surrounding context.

Method

The proposed experimental pipeline integrates three complementary components: a multilingual self-supervised speech backbone, supervised automatic speech recognition adaptation, and a separate measurement framework for the phoneme-to-orthography dependency of different writing systems. The first two components target ASR modeling, while the third quantifies a language-specific linguistic property that may help explain transcription difficulty.

The ASR backbone is Wav2Vec2-XLSR-53, which is built on the self-supervised speech representation paradigm of Wav2Vec 2.0. In the pretraining stage, a raw waveform $\mathcal{X}$X is passed through a convolutional feature encoder to produce latent speech representations $\mathcal{Z}$Z. These latent representations are then mapped to quantized representations $\mathcal{Q}$Q through product quantization. During training, some latent representation slots are masked with a fixed probability. A masked Transformer consumes these corrupted latent slots and predicts contextual representations $\mathcal{C}$C. The learning objective contrasts the predicted context representation with the correct quantized vector for the masked position and with incorrect quantized vectors, using a contrastive loss. This stage allows the model to learn speech representations without transcriptions.

As shown in the framework diagram:

For ASR fine-tuning, the pretrained model is adapted to each target language and writing system. The supervised objective is Connectionist Temporal Classification, which aligns variable-length acoustic frames with variable-length orthographic sequences. The same training configuration is reused across settings, including 20 epochs and a learning rate of $0.0003$0.0003. Punctuation is removed before training so that the model focuses on speech-to-text mapping rather than orthographic segmentation. Because the base model is self-supervised rather than weakly supervised with transcribed speech, the experimental setup reduces the possibility that fine-tuning performance is heavily biased by whether the target language already appeared in labeled pretraining data.

The pipeline also includes a controlled writing-system transformation module. Instead of only comparing languages with fixed native orthographies, the authors evaluate multiple transcription systems for the same language. For Japanese, the default Kanji and Kana orthography is compared with Kana-only and Romaji-only representations. Kana-only transcription is obtained through morphological tokenization and conversion, while Romaji-only transcription is generated by romanizing Kana. For Korean, the standard Hangul syllable representation is compared with Hangul Jamo, where each Hangul syllable is decomposed into phonemic components. For Chinese, the standard Hanzi representation is compared with Zhuyin and Pinyin. These conversions are performed deterministically during preprocessing, so differences in model performance can be related to grapheme inventory and character-level variability rather than to changes in the underlying speech data.

To characterize writing-system complexity, two grapheme-related measures are used. The first is the total number of distinct grapheme types observed in the training data. The second is the unigram character entropy of the training data, which accounts for unequal character frequencies. This entropy is computed as

where $C$C is the set of character types in the corpus. This entropy measure is important because many scripts do not distribute characters uniformly; instead, frequent characters dominate the distribution, and rare characters may increase the effective learning burden in ways that a simple type count may overstate or understate.

The second major component estimates logographicity, defined here as the degree to which a word’s orthography depends on surrounding context rather than being predictable from its pronunciation alone. To estimate this property, the authors train a separate phoneme-to-orthography sequence-to-sequence model. The model receives a short phonemic context and predicts the orthographic form of a target word enclosed by special tags. The input format includes a start symbol, surrounding phonemes, word-boundary markers, target-word tags, and an end symbol. The phonemic context contains seven tokens, with the target word in the middle and three phonemes to its left and right. The architecture consists of an encoder and a decoder. The encoder contains an embedding layer followed by a bidirectional GRU layer. The decoder contains an embedding layer, a unidirectional GRU layer, and a cross-attention layer that receives context vectors from the encoder. The attention matrix used for measurement comes from the cross-attention mechanism before residual connection and layer normalization.

The logographicity score is derived from how much attention remains after the attention region associated with the target word is removed. Let $A$A denote the attention matrix and $M$M denote a binary mask matrix of the same shape. The masked attention is written as the componentwise Hadamard product $M \circ A$M∘A. The mask sets selected entries to zero, suppressing attention mass that corresponds directly to the target word. The attention spread for a word $w$w is then

This quantity estimates the fraction of total attention retained after the target-word attention region is masked out. A lower retained attention indicates that the model relies primarily on the target word’s own pronunciation, which is expected for phonographic systems. A higher retained attention indicates greater dependence on surrounding context, which is expected for more logographic systems where pronunciation and spelling may be less directly mapped.

The corpus-level score averages the word-level attention spreads:

where $\mathcal{D}$D is the set of target words in the corpus. The masking operation and the resulting attention distributions are visualized below, where the original attention matrix is compared with the matrix after target-word attention values are suppressed.

The masking operation is visualized below:

Phonemization is handled through language-appropriate conversion tools. Japanese, Korean, and Chinese use the same conversion paths as the grapheme-transformation module, while other languages are converted into pronunciation representations using established phonemization resources or IPA conversion libraries. This preprocessing ensures that the logographicity measurement operates on comparable phonemic contexts across languages.

Experiment

The experiments evaluate automatic speech recognition with fine-tuned multilingual models across languages and alternate writing systems, especially Japanese, Korean, and Chinese, to test whether transcription accuracy is affected by orthographic complexity or phonological complexity. They use character-level evaluation and a time-calibrated error measure to compare different grapheme inventories and writing conventions. The qualitative findings show that logographic or large-grapheme writing systems make learning and transcription harder, leading to poorer and slower model convergence, whereas phonological complexity shows little clear adverse effect. The calibrated error measure also reduces differences caused purely by writing-system representation, supporting the view that orthography plays a major role in ASR performance.

The table compares datasets across languages with different script types, including logographic, syllabary, abugida, abjad, and alphabetic systems. The cited results suggest that more logographic writing systems and larger grapheme inventories are associated with harder automatic speech recognition behavior and slower learning. Logographic and mixed orthographies are associated with greater difficulty for speech-to-text transcription than phonographic systems. Complex writing systems show less smooth learning behavior and require more training progress than simpler orthographies.

Across the writing systems, larger grapheme inventories and more logographic or syllable character encodings are associated with higher character error rates and less stable learning. The phoneme type count does not show a significant relationship with accuracy, while CEPS shows weaker alignment with orthographic complexity than CER. English is reported to have markedly lower error rates, which may reflect stronger pretraining exposure and a cleaner read speech dataset. Phonographic alternatives such as kana, romaji, Hangul Jamo, and Zhuyin have lower error rates than corresponding complex character systems. CEPS reduces the apparent penalty from writing system representation, such as Hangul syllables versus Hangul Jamo. Orthographic complexity, not phonological inventory size, explains more of the observed ASR accuracy variation.

Character error rate is positively related to orthographic complexity, with stronger associations for grapheme inventory size, unigram entropy, and token-based logographicity. By contrast, the number of phoneme types shows no meaningful relation to character error rate, suggesting that the evaluated pretrained model remains relatively robust to phonological variation. The CEPS metric is correlated with character error rate but shows weaker links to orthographic complexity, indicating that it reduces the apparent impact of writing-system representation differences. Greater orthographic complexity is associated with higher character error rate. Phoneme inventory size does not appear to explain ASR error differences. CEPS correlates less strongly with orthographic complexity than CER does.

For Latin-script languages with phonographic writing systems, ASR error varies across languages, with some languages easier to transcribe than others. The phonographic metric tends to follow the same broad ordering as CER, indicating that it reflects relative transcription difficulty rather than phonological complexity. Among these languages, differences in grapheme inventory, entropy, and phoneme types do not appear to create a clear consistent relationship with performance. Basque shows much lower error than Kabyle and Indonesian in this Latin-script set. Kabyle has the highest CER among the listed languages, despite its Latin alphabet. CEPS values generally parallel CER rankings across the phonographic Latin-script languages.

For phonographic languages, character error rate shows only weak positive associations with orthographic measures such as grapheme count and unigram entropy. The CEPS score is strongly associated with character error rate, suggesting that it reflects similar difficulty patterns across these languages. By contrast, phonological complexity is not substantially related to transcription errors in this setting. CER and CEPS are strongly correlated. Orthographic measures have weak positive correlations with CER. Phonological complexity does not show a substantial correlation with CER.

The experiments evaluate how writing system type and orthographic complexity affect automatic speech recognition across languages. They find that logographic, mixed, and larger inventory scripts are generally harder to transcribe and show less stable learning, while phonographic systems tend to be easier. Overall, orthographic complexity explains performance variation more than phoneme inventory size, and CEPS closely tracks relative difficulty while reducing apparent differences caused by script representation.