One-sentence Summary

Researchers at the University of British Columbia, the University of Waterloo, and the Vector Institute introduce LINGGYM, a benchmark that evaluates LLMs' metalinguistic reasoning through a Word-Gloss Inference task on Interlinear Glossed Text drawn from 18 typologically diverse grammars, finding that structured linguistic cues consistently improve reasoning performance but that models still struggle to generalize to unseen low-resource languages and structures.

Key Contributions

• LINGGYM is a benchmark built from Interlinear Glossed Text and grammatical descriptions in 18 typologically diverse reference grammars to evaluate LLM metalinguistic reasoning across unseen, low-resource languages.
• The Word-Gloss Inference task requires models to infer missing words and glosses from context while controlling access to glosses, grammatical explanations, and translations.
• Experiments with multiple state-of-the-art LLMs show that structured linguistic cues consistently improve reasoning performance and reveal capabilities and limitations for typologically informed language documentation.

Introduction

In language documentation, linguistic hypothesis testing, and typological research, large language models could support analysis of morphology, syntax, and word order across diverse languages. However, most model evaluations remain centered on English and high-resource data, while prior low-resource NLP tools are often task-specific, limited to particular languages, or tested under overly clean and artificial conditions. The authors leverage interlinear glossed text and explicit grammatical explanations from reference grammars to evaluate how well LLMs can perform meta-linguistic reasoning in low-resource languages. They contribute a hand-verified dataset from 18 endangered and low-resource languages, an evaluation framework based on word-gloss inference with grammatical cues, and a typologically informed analysis of state-of-the-art LLM capabilities and limitations.

Dataset

• Source and composition: The authors build the benchmark from 18 publicly available reference grammars published by Language Science Press, selected from the Studies in Diversity Linguistics and Comprehensive Grammar Library series. The books are released under CC BY 4.0, and the LaTeX source files are retrieved from public GitHub sites.

• Language coverage and scale: The processed benchmark spans 18 grammars from 8 language families, with languages mostly from African and Pacific regions. After filtering and cleaning, it contains 19,612 interlinear glossed text examples aligned with relevant grammatical knowledge points.

• Subset and subfield details: Chapters are manually classified into phonology, morphology, syntax, semantics, pragmatics, or other linguistic subfields. The benchmark covers major grammatical description areas, with a strong emphasis on syntax, reflecting common language documentation practices.

• Inclusion and filtering rules: The authors keep chapters that contain labelled sections describing grammatical rules or linguistic content and that include interlinear glossed examples linked to those rule explanations. Examples must include a bolded target keyword, such as a word, morpheme, or form under discussion. Chapters without usable interlinear examples, such as acknowledgments or appendices, are excluded, and phonetics chapters are excluded because of sparse interlinear content and inconsistent International Phonetic Alphabet formatting.

• Extraction and cleaning: LaTeX files are converted to plain text, removing formatting commands while preserving bold keywords that mark grammatical focus. For each example, the authors extract structured interlinear fields, including transcription where applicable, morpheme segmentation, glosses, and free translation, together with preceding knowledge point paragraphs that explain the grammatical rationale. They also record hierarchical metadata: chapter title, section heading, and subsection heading.

• Structural validation: After automated extraction, examples are manually cleaned to ensure complete and aligned interlinear glossing. The number of words must match the number of glosses, and within each word, hyphen-separated morphemes must align one-to-one with glosses. The morpheme-segmented line is used as the standardized language input because it provides finer-grained grammatical units and better gloss alignment.

• Data schema: Each benchmark item centers on an aligned linguistic example and may include the morpheme-segmented source sentence, gloss information, knowledge point, English translation, and hierarchical source metadata.

• Question construction: The authors create cloze-style word or word-gloss inference questions from cleaned examples. The target is identified by a bold tag. One correct answer and three distractors are generated using three strategies: a form-based distractor sharing the longest common substring with the correct gloss, a semantics-based distractor selected by Sentence-BERT similarity, and a chapter-local distractor sampled from the same grammar chapter. Correct answer positions are randomized uniformly across four options.

• Difficulty levels and usage: The authors use the dataset as an evaluation benchmark rather than as a training corpus. No training split or mixture ratios are reported. Evaluation prompts vary by available information: surface sentence only, surface plus gloss, plus knowledge point, and plus English translation.

• Cropping and other processing: No cropping strategy is described. The key processing steps are LaTeX-to-text conversion, structural filtering, manual chapter categorization, interlinear alignment validation, metadata construction, target masking, and distractor generation.

Method

The method is built around a cloze-style, multiple-choice word/gloss inference task derived from cleaned interlinear glossed text examples. Each example is transformed into a prompt in which one target element is removed. The target may be a single surface word, a single word-gloss pair, or the surface form required by the missing gloss. The model must recover the missing element from four alternatives by combining evidence from the segmented sentence, morphological glosses, available grammatical explanation, and optionally the English translation. This setup evaluates whether large language models can infer grammatical information from structured linguistic data rather than merely recognize frequent surface forms. The end-to-end transformation from IGT data to a question-answer evaluation is shown in the figure below.

The question construction process begins by identifying a target word in each example. If the word itself or any of its morphemes carries a special annotation in the source representation, that word is selected as the masked target. The pipeline then produces one correct candidate and three distractors. The first distractor strategy is form based. It selects a gloss that shares a long lexical or morphological substring with the correct gloss but changes the grammatical function. For example, a correct word-gloss association involving a root meaning walk with a past tense gloss can be contrasted with an alternative that keeps the same root but uses a progressive gloss. The second strategy is semantic. Glosses are embedded using Sentence-BERT, and the most semantically similar but nonidentical gloss is mapped back to a word in the dataset. This creates choices that differ in fine-grained meaning rather than only in surface form. The third strategy is chapter-local. It samples a word-gloss pair from the same grammar chapter while ensuring that the candidate does not overlap with the correct answer or with the other distractors. This preserves topic relevance while avoiding overly simple elimination.

After candidate generation, the system enforces uniqueness across all four options and randomizes the position of the correct answer. The correct option is placed uniformly among the four answer slots, so that each position corresponds to the correct answer approximately 25% of the time. This prevents models from exploiting fixed answer distributions. The prompt is then constructed by masking the relevant target element. Masking in the surface line is performed at the word level to keep granularity consistent. Masking is also applied to the gloss line and to any knowledge point line that mentions the target word or gloss. This preserves the local context while clearly marking the missing information. The English translation line is more difficult to mask because it often paraphrases the target or uses a semantically related expression rather than a direct lexical equivalent. As a result, the surface and gloss lines are systematically controlled, while the translation may still provide indirect cues about the answer.

The benchmark further controls the amount of linguistic information exposed to the model through four difficulty levels. The simplest input contains only the original morpheme-segmented sentence, denoted $S$S. The next level adds gloss information, giving $S+G$S+G. The third level adds relevant grammar knowledge points, giving $S+G+KP$S+G+KP. The richest level also adds the English translation, giving $S+G+KP+T$S+G+KP+T. These conditions isolate whether grammatical reasoning improves when more structured or cross-linguistic information is available. The shared prompt template for these variants includes a linguistic role setting, a masked target language sentence, a masked gloss line, an optional translation field, an optional knowledge point field with related morphemes and glosses masked, four word-gloss options, and an instruction to return only the letter of the selected answer. The complete prompt format is summarized in the framework diagram.

Experiment

The experiments evaluate a multiple-choice word-gloss inference benchmark built from interlinear glossed examples drawn from reference grammars of many underrepresented languages. Across several open-source model families and increasing prompt contexts, the results show that language models can perform meaningfully above chance on specialized morphosyntactic reasoning, with knowledge points, glosses, and translations each contributing useful evidence. Performance is relatively stable across languages and linguistic subfields, but remaining errors reveal difficulties with dense gloss abbreviations, semantically plausible distractors, and fine-grained phonological or orthographic distinctions.

The LINGGYM dataset aligns knowledge points with interlinear glossed text from reference grammars across eight language families, with substantial representation of underrepresented languages from African and Pacific regions. Example counts vary widely by language, with Papuan Malay, Pichi, and Rapa Nui standing out among the listed entries. The dataset is used to analyze model failures that involve opaque gloss abbreviations, semantically similar distractors, and minimal morphological or orthographic differences. Coverage spans eight language families, with several Atlantic-Congo and Austronesian languages included. The listed language counts suggest that a few grammars contribute a larger share of aligned examples than others. Even strong model configurations can fail when gloss tags encode dense grammatical information that must be parsed rather than guessed. Model errors often arise from choosing forms that are plausible in English translation but do not satisfy the target language morphology.

The examples are distributed across linguistic subfields, with syntax forming the largest share by a wide margin. Morphology and semantics appear as smaller but notable categories, while phonology, pragmatics, and other topics are much less represented. This pattern reflects a strong focus on syntactic description, with broader coverage across other areas of linguistic structure. Syntactic examples dominate the distribution. Coverage spans multiple linguistic subfields, with phonology and pragmatics represented most sparingly.

Accuracies increase steadily across model families as glosses, knowledge points, and translations are added, with the most complete prompt setting producing the strongest results. Even raw sentence-only inputs exceed chance for every model, suggesting some pretraining exposure while leaving the benchmark far from saturated. The strongest DeepSeek-R1 model performs best, but remaining failures concentrate on opaque gloss tags, semantically plausible distractors, and minimal form differences. Raw sentence-only inputs already exceed chance across all evaluated models. Knowledge points consistently improve accuracy across model families and parameter scales. Larger and reasoning-oriented models generally outperform smaller models, with DeepSeek-R1 32B strongest. Persistent errors involve uninterpreted gloss abbreviations, morphosyntactically wrong synonyms, and fine-grained orthographic differences.

The selected models show a broad improvement when supplementary linguistic information is added to the raw sentences, especially when translations are included. Knowledge points and glosses also raise performance, but the gains vary by model, with Gemma3-12B generally reaching the highest accuracies in translation-inclusive combinations. The pattern suggests that LLMs can use partial contextual cues, yet even the best configurations leave substantial room for improvement on this specialized meta-linguistic benchmark. Sentence-only inputs already yield above-chance performance, indicating possible memorization but not full mastery. Adding translations produces strong gains, particularly for Gemma3-12B and several smaller models. Combinations with knowledge points, glosses, or translations improve accuracy over the raw-sentence baseline, but no configuration approaches perfect performance.

The experiments evaluate models on the LINGGYM benchmark, which covers underrepresented languages across multiple language families and linguistic subfields, using raw sentences and progressively richer prompts with glosses, knowledge points, and translations. The results show that models are already above chance from pretraining exposure, while added linguistic context generally improves performance, especially translations and reasoning-oriented or larger models. However, persistent failures on dense gloss abbreviations, morphosyntactically plausible but incorrect forms, and subtle orthographic differences indicate that reliable meta linguistic reasoning over specialized grammars remains unsolved.