One-sentence Summary

Researchers from UC San Diego and Okalai AI introduce MoI-MoE, a Mixture of Experts model that leverages the Morphological Index (MoI) to route inputs by morphological structure, improving Swahili noun class recognition by $14$14 across $10$10 while preserving performance on morphology-rich languages like Zulu.

Key Contributions

• The paper introduces the Morphological Index (MoI), a typologically grounded metric that quantifies how strongly a language relies on surface morphology for noun classification, reframing language diversity as structural variation rather than data imbalance.
• Building on MoI, the paper proposes MoI-MoE, a Mixture of Experts model that routes inputs by morphological structure. This treats typology as an architectural design principle, guiding expert selection and parameter sharing among structurally aligned languages.
• Evaluated on 10 Bantu languages, MoI-MoE outperforms monolithic and token overlap baselines on noun class recognition, improving Swahili accuracy by 14 points while maintaining performance on Zulu. It also supports downstream POS tagging and offers a scalable solution for labeling nouns in dictionaries lacking noun class annotations.

Introduction

The authors tackle a structural weakness in multilingual modeling: monolithic transformers like mBERT and XLM-R share parameters across typologically diverse languages, which dilutes representational capacity and leads to poor performance on African low-resource languages. Prior specialization efforts, such as AfriBERTa and BantuBERTa, group languages by genetic lineage, but this is too coarse to capture functional differences; for example, Swahili and Zulu are both Bantu yet diverge sharply in morphological surface structure. The authors propose that structural typology, rather than ancestry, should guide model grouping.

To operationalize this, they introduce the Morphological Index (MoI), a continuous, interpretable metric that quantifies how much a language relies on morphology for noun classification. They then build MoI-MoE, a Mixture of Experts model where each expert specializes in a distinct morphological regime, with a lightweight, non-learned router selecting experts based on MoI. This typology-driven routing amplifies semantic features in low-morphology languages while maintaining accuracy in high-morphology ones. The authors validate MoI through three analyses: correlation with rule-based accuracy, interaction with semantic versus morphological features, and routing performance. They also define Noun Class Recognition (NCR) as a core evaluation task, releasing datasets covering 65K nouns across 10 Bantu languages. Prior typology-aware work treats typology as metadata, using coarse language vectors from databases like WALS, which are too sparsely populated to distinguish structurally similar languages; this work instead treats typology as an architectural design principle, promoting transfer among structurally aligned languages and reducing cross-linguistic interference.

Dataset

The authors construct two datasets for noun class recognition (NCR) in Bantu languages. Each entry contains a Bantu noun, its class label, an English gloss, and a GPT-4-expanded definition.

Dataset composition and sources

• DictionaryNCR is the primary dataset, with roughly 63,000 nouns across 10 languages. It is assembled from bilingual dictionaries and grammar books obtained via Google Books, the Library of Congress, university archives, and online sources.
• WikiNCR is a smaller out-of-domain test set with about 2,600 nouns across 3 languages, sourced from Wiktionary, where certain languages provide explicit noun class labels (parsed from HTML entries using BeautifulSoup).

Key details per subset

• DictionaryNCR sources fall into two types:
  • Type 1 (explicit class labels): Dictionaries for Swahili, Kwanyama, Shona, Luganda, and Sotho annotate noun classes directly. These entries (31,524 of 62,844) required only extraction and formatting.
  • Type 2 (inferred class labels): Dictionaries for Ndonga, Umbundu, Xhosa, Zulu, and Herero provide nouns with prefixes but no explicit class labels. The authors infer class labels from prefixes using grammar-derived mappings, resolving ambiguous cases heuristically via prefix patterns and glosses.
• WikiNCR has limited coverage but serves as a complementary testbed for out-of-domain evaluation.

Processing and filtering

• Most DictionaryNCR sources were scanned or in PDF form, so the authors applied OCR via MathPix, followed by parsing and structuring with GPT-4.
• Manual inspection of 200 samples across two languages revealed no hallucinated content, indicating good quality of the extraction pipeline.

Usage in the paper

• DictionaryNCR is the primary training and evaluation dataset, while WikiNCR functions as an out-of-domain test set to assess generalization.
• Noun class distributions are reported for both datasets in the appendix (Table 9 for DictionaryNCR, Table 10 for WikiNCR), and overall statistics are summarized in Table 3.

Method

The authors propose a method that leverages the Morphological Index (MoI) to guide the design of a multilingual model for Noun Class Recognition (NCR). The MoI quantifies how explicitly noun class distinctions are marked in a language's surface morphology by summing the character lengths of all noun class prefixes.

Refer to the framework diagram illustrating the decomposition of Bantu nouns into prefixes and stems.

The index is computed as:

where $\mathcal{C}_l$Cl​ represents the set of noun classes in language $l$l, and $|p_c|$∣pc​∣ is the character length of the prefix for class $c$c. This metric allows the system to distinguish between High-MoI languages, which rely on longer, distinctive prefixes, and Low-MoI languages, where shorter or overlapping prefixes shift the classification burden to semantics. The distribution of these scores across languages reveals a clear separation, with a cutoff value of 50.0 used to stratify the languages into distinct morphological regimes.

As shown in the figure below, languages are plotted based on their MoI scores, demonstrating the gradient of morphological dependence.

To process these inputs, the model utilizes two primary feature sets. Morphological features are extracted using SentencePiece tokenization, allowing the model to learn subword patterns that correlate with noun classes without explicit supervision. Semantic features are introduced by pairing each Bantu noun with its English gloss during training. This approach grounds the predictions in a shared semantic space via pretrained English embeddings, compensating for the sparse coverage of Bantu languages in large language models.

The base model is built upon mT5-XL, which is fine-tuned using a supervised sequence-to-sequence objective. The training prompt follows the structure: "Given the Bantu noun [X] whose English meaning is [Y], predict its noun class," where [Y] includes an expanded definition. The model is optimized using a standard cross-entropy loss:

To ensure robust generalization across languages with varying resource levels, balanced language sampling is applied during training.

The core architectural innovation is the MoI-MoE (Mixture of Experts) framework, which adapts the model based on the typological profile of the input language. This architecture employs a lightweight, nonparametric router that directs inputs to specialized experts based on their MoI scores.

Refer to the architecture diagram showing the routing mechanism.

The system defines two distinct experts to handle different morphological regimes. The High-MoI Expert is optimized for languages where subword token patterns approximating prefixes and stems provide strong morphological signals. Conversely, the Low-MoI Expert is designed for languages where prefix forms are less informative, requiring the model to rely more heavily on semantic context provided by the English glosses. This structure-aware routing allows the model to generalize across typologically similar languages, even when they differ significantly in vocabulary.

Experiment

Experiments on eight Bantu languages validate MoI as a linguistically grounded measure of morphological salience: rule-based classifier accuracy tracks MoI (r = 0.75), while semantic feature gains negatively correlate with it (r = -0.736), confirming that low-MoI languages rely on meaning while high-MoI languages can depend on form alone. MoI-guided routing in a mixture-of-experts model (MoI-MoE) achieves 92% accuracy, outperforming a token-overlap baseline (80%) and a monolithic model (89.9%), with ablations confirming that subword tokenization and noun class prefixes are essential, and probing showing MoI encodes structure beyond augment presence. Error analysis reveals remaining mistakes track structural ambiguity, such as overlapping prefixes in high-MoI languages and ambiguous nasal prefixes in low-MoI ones. On generalization, MoI-based routing improves zero-shot transfer to unseen languages, though cross-dataset performance drops highlight overfitting to dictionary-style input, and integrating noun class distinctions into POS tagging causes only a modest 1.0% accuracy loss.

The table shows that Bantu languages share cognates for basic concepts but realize them with different noun class prefixes, indicating that phonological similarity does not guarantee structural similarity. This motivates the Morphological Index (MoI), which quantifies how explicitly noun class distinctions are marked. MoI-guided modeling is shown to be sensitive to these structural variations, affecting error patterns and generalization. Cognates for words like snake and finger share roots but differ in noun class prefixes across languages. High-MoI languages use longer, more distinctive prefixes, while low-MoI languages rely more on semantics due to shorter or overlapping prefixes. Errors in high-MoI languages often stem from overlapping prefixes, whereas low-MoI languages show semantically driven confusions.

The table compares noun class prefixes in Swahili and Xhosa, revealing that while both languages use a similar class system, their prefix forms differ notably in length and variety. Xhosa generally employs longer, more distinctive prefixes, whereas Swahili uses shorter, often overlapping forms, illustrating how morphological marking varies across Bantu languages. Xhosa prefixes are consistently longer (e.g., um-, aba-, isi-) than Swahili's (m-, wa-, ki-), indicating a higher degree of morphological distinctiveness. Swahili uses the same prefix m- for Classes 1 and 3, while Xhosa does the same with um- for both, but Xhosa also shows more allomorphic variation such as aba-/abe- for Class 2 and iiN-/iziN- for Class 10.

The table presents statistics for two datasets: DictionaryNCR, a large resource for training and development across several Bantu languages, and WikiNCR, a smaller out-of-domain test set for a subset of these languages. DictionaryNCR provides substantial training and validation splits for each language, with Swahili having the largest sample, while WikiNCR only covers three languages with much smaller sizes. The datasets differ in scope and purpose, supporting both model development and generalization testing. DictionaryNCR contains over 60,000 total nouns across multiple languages, with Swahili and Ndonga contributing the largest training splits. WikiNCR is limited to Swahili, Xhosa, and Zulu, and is significantly smaller than the corresponding DictionaryNCR splits. Language-specific sizes vary widely, reflecting uneven availability of linguistic resources.

Bantu noun classes map to broad semantic categories, such as humans in classes 1/2, plants in 3/4, and animals in 9/10, alongside other groupings like abstracts and infinitives. This semantic regularity supports using English glosses as a proxy for meaning in modeling noun class recognition. Human nouns typically fall into classes 1/2, while plants and trees appear in classes 3/4. Classes 9/10 cover animals and some inanimates, and classes 5/6 include paired objects, liquids, and masses. Other classes cover abstracts, mass nouns, and infinitives, showing a structured semantic organization across the class system.

MoI-MoE achieves the highest overall accuracy on Bantu noun class recognition, outperforming both a token-overlap-based MoE and a monolithic multilingual model. The improvement is especially pronounced in low-MoI languages like Swahili, where semantic features add substantial value, while high-MoI languages like Zulu show little difference, indicating morphology alone suffices. Error analysis reveals that remaining errors stem from structural ambiguity, with prefix overlaps and semantic confusions varying by MoI level. MoI-MoE outperforms both the token-overlap MoE and the monolithic model across eight Bantu languages. Semantic features yield a large accuracy gain in Swahili, a low-MoI language, but almost no gain in Zulu, a high-MoI language. Errors in high-MoI languages are dominated by overlapping prefixes, while low-MoI languages face more semantically driven confusions.

Experiments on Bantu noun class recognition used the Morphological Index (MoI) to quantify morphological explicitness, with a MoI-guided mixture of experts (MoI-MoE) model compared against baselines. The model performed best overall, with the largest gains in low-MoI languages like Swahili, where semantic features compensate for overlapping prefixes, while high-MoI languages like Zulu rely largely on morphology. Error analysis revealed that high-MoI errors stem from prefix overlaps, whereas low-MoI errors are semantically driven. Datasets varied in size and domain, with an out-of-domain WikiNCR test for generalization.