One-sentence Summary

University of Colorado Boulder and Carnegie Mellon University propose POLYGLOSS, a family of multilingual seq2seq models that jointly predict interlinear glosses and morphological segmentation from raw text, addressing the inability of prior models like GLOSSLM to predict morpheme boundaries, outperforming GLOSSLM on glossing and beating open-source LLMs on segmentation, glossing, and alignment while supporting efficient low-rank adaptation to new datasets.

Key Contributions

• POLYGLOSS, a family of multilingual seq2seq models, is introduced to jointly predict interlinear glosses and morphological segmentation from raw text in a single pass, addressing the lack of interpretable morpheme boundaries in prior glossing systems.
• An extended and standardized GLOSSLM corpus of 341k examples is released. Models trained on this corpus outperform GLOSSLM on glossing and surpass various open-source LLMs on segmentation, glossing, and alignment.
• The work demonstrates practical deployment paths by showing that POLYGLOSS can be rapidly adapted to new datasets through low-rank adaptation. It also shows that per-language perplexity can approximate glossing quality for filtering or fallback.

Introduction

Language documentation is increasingly urgent because nearly half of the world’s languages face extinction. Interlinear glossed text combines morphological segmentation, tags, and translation, supporting linguistic analysis, teaching, and tools such as searchable corpora and machine translation. Prior automated glossing systems mainly predict glosses from transcription or segmentation, but user studies show that direct gloss output can be hard to interpret, unreliable for some languages, and mismatched with preferred glossing conventions. The authors leverage an expanded and standardized GLOSSLM corpus of 341k examples and train a multilingual model that jointly predicts morphological segments and glosses. They also show that per-language perplexity can estimate glossing quality for filtering or fallback, and that low-rank adaptation can quickly adapt model outputs to new label conventions.

Dataset

• The authors construct an enhanced version of the GLOSSLM corpus by improving the original corpus formatting and adding new interlinear glossed text data. The base data remain aligned with GLOSSLM, while the enhanced corpus adds cleaner processing and larger evaluation coverage.

• Additional IGT data comes from two main sources. The Fieldwork dataset contributes 80,461 IGT instances across 37 languages. IMTVault is updated to version 1.2, adding 39,741 examples scraped from newer linguistic publications.

• The authors remove 20,116 duplicate examples and filter very low quality examples from these additions. The final additional data amount to 91,416 unique examples compared with the original GLOSSLM corpus.

• Punctuation is normalized across all sources. Sentence final punctuation is surrounded by spaces, while punctuation inside glosses is preserved unchanged. This supports more consistent segmentation, tokenization, and gloss alignment.

• Source specific formatting errors are corrected. For example, in the Arapaho data, 4,882 instances containing the sequence ",." inside glosses were identified and confirmed by the original annotator as errors.

• The authors audit segmentation and gloss alignment. If a morphological segmentation field has no segmentation markers but the gloss field has multiple segments, the segmentation is set to blank. If mismatched examples retain segmentation, they are placed only in the training split so they do not affect evaluation.

• The dataset is used for both training and evaluation. Misaligned or problematic examples are either cleaned, filtered, or routed into the training split, leaving evaluation data more reliable.

• Two new evaluation languages are added from the Fieldwork dataset, Hokkaido Ainu and Ruuli. The paper also provides split information for evaluation languages, along with quality and issue statistics from the auditing process.

Method

The authors construct POLYGLOSS as a continued pretraining framework for multilingual interlinear glossing. The base model is ByT5, a byte level encoder decoder transformer based on the T5 architecture. The byte level input representation is selected to avoid subword tokenization failures for rare or typologically distant languages. The model is trained from the byt5 base checkpoint with 580 million parameters. During continued pretraining, the model learns to predict glosses from transcriptions and, when available, from morphological segmentations. The preferred evaluation setting uses unsegmented transcriptions because this is the more difficult and realistic form of input.

The method is organized around three output formulations that differ in how they model the relationship between segmentation and glossing. In the multitask prediction setting, the pipeline creates separate training examples for segmentation and for glossing. This allows the system to infer both outputs from the same input model, but alignment between segments and glosses is not explicitly constrained. The authors identify this as a possible source of misalignment because the two outputs are generated from independently supervised targets.

To capture the dependency between segmentation and glossing, the pipeline also uses a concatenated prediction formulation. In this setting, the model is trained to output the morphological segmentation first and then the glosses within a single generated sequence. Because training follows a causal language modeling objective, the model can attend to the earlier segmentation tokens while generating the gloss string. This creates an implicit dependency from glosses to segmentation, although it does not guarantee strict alignment. A poorly generated segmentation can therefore propagate errors into the gloss sequence. To account for examples that contain gloss labels but not segmentation labels, the training mixture also includes separate glossing examples in the multitask style.

The most structured formulation is interleaved prediction, which introduces a hard alignment constraint into the output format. In this setting, every gloss label is immediately followed by its corresponding morpheme in parentheses. This arrangement forces generated outputs to expose the mapping between glosses and segments directly. The authors hypothesize that this format should produce the strongest alignment because a well formed prediction must contain paired gloss and morpheme tokens.

The training process keeps the optimization recipe relatively conservative. All models are trained in bf16 precision using AdamW with default hyperparameters. The learning rate schedule begins with linear warmup for the first 3% of training steps and then follows cosine decay. Gradient clipping is applied with a maximum norm of 1. The ByT5 variants are trained with a learning rate of $5 \times 10^{-5}$5×10−5, batch size 64, and 15 epochs. The setup uses four GH200 GPUs, and evaluation is performed with beam search using two beams.

For deployment to new languages or annotation styles, the authors adapt POLYGLOSS with low rank adaptation, or LoRA, rather than full parameter finetuning. This reduces computational cost and enables adaptation from small numbers of glossed examples. In a simulated annotation scenario for a language unseen during pretraining, rank 8 adapters are trained on increasingly large training sets in increments of 50 examples. The largest adaptation run uses 25 epochs, batch size 32, and requires only 12 minutes on an A100 GPU. The adaptation behavior for the interleaved POLYGLOSS model and a ByT5 base model is shown in the figure below.

The figure illustrates that the pretrained POLYGLOSS representation can be quickly adapted to a new language as more data becomes available, while the ByT5 baseline remains at a high morpheme error rate. This supports the design choice of starting from the corpus pretrained POLYGLOSS model and applying lightweight parameter efficient adaptation instead of training a general multilingual model from scratch for each new language.

The authors also connect the alignment scoring mechanism to a reinforcement learning posttraining process. The alignment score is used as a scalar reward function for reinforcement learning with verifiable rewards. In a pilot study on the concatenated prediction model, vanilla GRPO is used to improve output consistency. The optimization uses $\beta = 0.1$β=0.1 and learning rate $5 \times 10^{-5}$5×10−5. Batch size and group size are both set to 8, and the model is trained for 50 epochs. During sampling, the decoding temperature is 0.6, top p is 0.9, and a repetition penalty of 1.05 is applied.

The before and after results from this alignment reward optimization are shown in the figure below.

The figure reports mean gloss MER, segmentation F1, and alignment scores before and after GRPO tuning. The reward based update produces slight improvements across these metrics. This process can be interpreted as a posttraining stage that optimizes the model toward outputs that are easier to align, without requiring additional labeled data. It also provides a possible adaptation mechanism for new languages that lack gold morphological segmentations, because the alignment reward encourages generated segmentations to correspond to the available gloss labels.

Experiment

The experiments evaluate joint glossing and segmentation using glossing, segmentation, and alignment metrics, and compare the proposed model against pretrained baselines, large language model prompting, monolingual variants, and adaptation strategies. Results show that jointly modeling both tasks, especially in an interleaved format, produces accurate and well aligned outputs, while multilingual pretraining offers transfer benefits for languages with limited data. The alignment score also supports practical use by helping predict performance on new languages and enabling further improvement through adaptation and reinforcement learning.

The statistics describe the final POLYGLOSS corpus, which combines cleaned GLOSSLM data with additional interlinear glossed text sources. The corpus covers over two thousand languages, with most examples assigned to the training split and smaller evaluation and test sets retained. A minority of examples lack language or metalanguage glottocodes, indicating incomplete metadata in some sources. The corpus has broad multilingual scale, with hundreds of thousands of examples across thousands of unique languages. The train split dominates the data, while evaluation and test splits are comparatively small but separate. Some examples remain unclassified in language or metalanguage metadata, reflecting source limitations.

The split information shows that the evaluation corpus is highly uneven across languages, with Arapaho having the largest training and evaluation/test resources and Gitksan having very small train, eval, and test sets. Most languages have training sets much larger than their eval and test sets, indicating that data availability is a major difference among the languages included. Arapaho is the best represented language, with markedly more training data than other languages and also the largest eval and test portions. Gitksan is the least represented language, with very limited training examples and small evaluation and test sets, making it a low-resource case in the corpus. Some languages, including Uspanteko and Ainu, have larger test sets than eval sets, while still having much smaller eval and test data than their training data.

The results compare glossing scores for multilingual models across nine held out test languages. Across the displayed rows, POLYGLOSS ByT5 interleaved has the highest overall average, while Qwen 0.6B with in context learning has the lowest overall average. The cited explanation describes POLYGLOSS as outperforming the in context learning baselines, and notes that GLOSSLM is poor on languages outside arp, ddo, and git. POLYGLOSS ByT5 interleaved has the highest overall average among the listed models Qwen 0.6B has the lowest overall average among the listed models The explanatory text attributes some Qwen 0.6B issues to format conformance and alignment failure

The interleaved POLYGLOSS ByT5 model reaches a perfect score on every language shown, indicating very strong held-out multilingual performance. Its multitask ByT5 variant remains strong overall, but it does not reach the perfect result of the interleaved format. In-context learning baselines are more uneven, with the smallest Qwen model weakest overall and larger LLMs achieving high yet imperfect scores. Interleaved POLYGLOSS is perfect on all shown languages. Multitask POLYGLOSS is strong but weaker than the interleaved variant on the average. The smallest Qwen model is the weakest in-context baseline, while the larger Gemma and Aya models are stronger but still imperfect.

The experiments evaluate the constructed POLYGLOSS corpus, which spans many languages but contains uneven data across languages and splits. Held-out glossing tests across nine languages show that the interleaved POLYGLOSS ByT5 model is the strongest, reaching perfect scores on the displayed languages, while the multitask ByT5 variant remains competitive but less robust. In context learning baselines are more unstable, with the smallest Qwen model performing weakest because of format and alignment failures, indicating that dedicated modeling is more reliable than prompting alone for gloss generation.