One-sentence Summary

Researchers at The University of Texas at Austin demonstrate through counterfactual corpus experiments that transformer language models learn the rare English Article+Adjective+Numeral+Noun construction (e.g., "a beautiful five days") by generalizing from less rare related constructions such as "a few days," providing an existence proof that generalization rather than memorization drives the acquisition of rare grammatical phenomena.

Key Contributions

• BabyLM-trained transformer language models learn the English Article+Adjective+Numeral+Noun construction from human-scale, systematically manipulated corpora and judge novel AANNs well above chance even without AANN exposure, while models trained on word-order shuffled variants generalize less effectively.

• Counterfactual corpora show that generalization to unseen AANNs depends on related constructions, because holding out singular treatment of measure noun phrases such as "a few days" lowers unseen AANN likelihood by 36.5% on average, and data-volume controls indicate the decline reflects loss of structurally related evidence rather than smaller training sets.

• Variability in rare-construction inputs improves generalization, with models trained on AANNs containing more varied adjective, numeral, and noun slots outperforming models trained on restricted repeating instances, supporting statistical-learner accounts of rare phenomenon acquisition.

Introduction

Humans can accept and produce rare grammatical constructions, such as “a beautiful five days” or the Participle Island Protection Property pattern, even when such structures are virtually absent from experience. Whether large language models can similarly learn rare grammar from realistic amounts of data matters because it tests whether statistical learning and general abstraction can explain linguistic knowledge without strong innate biases. Prior work is limited because modern LLMs often see vastly more data than humans do, making it hard to tell whether rare constructions are memorized or truly abstracted, and controlled-pretraining studies have mostly focused on common structures rather than the linguistic long tail. The authors leverage the 100M-word BabyLM corpus and systematic data manipulation to train transformers on a rare adjective-adjective-numeral-noun construction while holding out or altering related frequent phenomena. They show that models generalize to novel rare constructions even without direct exposure, especially when related constructions, such as measure phrases behaving like singular noun phrases, are present in training, providing evidence that rare grammar can emerge from abstraction over more frequent structures.

Dataset

• Base training corpus: The authors use the BabyLM-strict corpus as the main training set. It contains about 100 million tokens and is chosen because its size is human-realistic yet tractable enough for large-scale modeling and careful inspection of the target construction.

• Target-instance detection in the training corpus: To identify AANN constructions in BabyLM, the authors apply a regular expression to a part-of-speech-tagged version of the corpus. They refine the regex using hand-annotated examples selected with a very permissive search pattern, achieving high estimated recall. The final regex finds 2,448 AANN instances, roughly 0.02% of the 11.5 million utterances.

• Recall-robustness controls: Because regex-based detection cannot guarantee perfect recall, the authors also create a conservative control that assumes 300 missed AANN instances and artificially pollutes the data with them. This checks whether the findings depend on undetected instances.

• Evaluation corpus: The test data come from Mahowald (2023), which contains 12,960 templatically generated sentences with AANN constructions. Of these, 3,420 include human acceptability ratings from 190 participants on a scale from 1 to 10.

• Evaluation filtering: The authors keep only items with average human acceptability ratings above 7. They also remove 4 instances that appear in the BabyLM training set to avoid contamination. This yields a final evaluation set of 2,027 items.

• Metadata and alternative formulations: Each evaluation item includes the target AANN sentence plus corrupted variants that manipulate adjective-numeral order, article presence, adjective presence, and numeral presence. These variants let the authors assess whether models assign higher likelihood to well-formed AANN sentences than to systematically corrupted versions.

Method

The authors leverage a compact autoregressive language modeling pipeline that combines model training, targeted construction detection, and acceptability based scoring. The core model is an OPT style transformer language model. It uses 12 transformer layers and 12 attention heads, a vocabulary of 16,384 subword tokens, and training for up to 20 epochs with the transformers library. Reported estimates average over three independent runs with different random seeds to reduce variance.

A detection module identifies instances of the target AANN construction in the training corpus. The workflow operates on a version of the corpus tagged with part of speech information. It starts with a broad regex that searches for sequences resembling an article followed by a numeral and a plural noun, then uses hand annotated feedback to iteratively tighten the rule. This creates a practical filtering workflow for isolating the targeted construction while reducing false positives. Because rule based detection may still miss rare instances, the authors include a robustness control that assumes a conservatively high number of undetected AANNs and artificially pollutes the data with synthetic occurrences. This step allows the pipeline to test whether the observed effects remain stable under imperfect construction detection.

For evaluation, the authors use data containing human acceptability ratings from template generated sentences. They keep only examples where human participants rated the construction as clearly acceptable, using a threshold above 7, and they discard examples that also appear in the training corpus. The remaining examples are paired with corrupted alternatives that modify adjective numeral order, article presence, adjective presence, and numeral presence. This creates a forced choice test in which a model with knowledge of the target construction should prefer the well formed version over each corrupted variant.

The model is scored with the Syntactic Log odds Ratio. Given a language model $m$m, a unigram estimator $u$u, a target construction $\mathcal{C}$C, and a prefix, the score is defined as:

The unigram estimator is trained on the same corpus using the same tokenizer as the autoregressive model. This normalization reduces the chance that accuracy differences are driven only by surface frequency changes introduced by the data manipulations. Log probabilities are computed using the minicons utility. An item is considered correct if the SLOR value of the well formed construction is higher than the SLOR value of all four corrupted variants. Accuracy is the proportion of such correct items, and chance performance for the five option comparison is 20%.

Experiment

In a controlled BabyLM setup with token matched ablations, the experiments test whether language models learn the rare AANN construction, what surrounding phenomena support that learning, and how instance variability affects productivity. The models accept AANNs above chance despite little or no direct evidence, prefer grammatical AANNs over counterfactual word orders, and track known lexical constraints. Ablations show that related constructions, frequency balancing around articles, and measure noun singular treatment are critical when no AANNs are seen, while direct exposure plus high slot variability further improves generalization, indicating reliance on systematic distributional and constructional cues rather than shallow statistics.

The examples contrast the well-formed AANN pattern with two counterfactual variants, ANAN and NAAN, using the same lexical items. They illustrate how each construction can be minimally corrupted by changing order or removing the article, adjective, or numeral. The well-formed AANN example and its corruptions support tests of whether models prefer grammatical order over altered versions. Corruption examples include adjective-numeral reordering and missing articles, modifiers, or numerals.

The experiment examines whether language models learn the AANN construction from other constructions that share features with it, such as plural phrases following determiners, few/couple/dozen noun phrases, and measure noun phrases with singular verbs or indefinite articles. The frequency profile suggests that possible analogical evidence is more common than direct AANN examples, and slot variability tests show that highly variable adjective and noun fillers support broader generalization than restricted fillers. Removing these related constructions lets researchers test their role in unseen AANN acceptability. Models exposed to highly variable AANN slots, except the numeral slot, show greater productivity.

The experiments compare the AANN construction with counterfactual and corrupted variants using the same lexical items to test whether models prefer well-formed order and structure. They also examine whether related constructions, such as plural and measure noun phrases, provide analogical evidence for learning AANN, while slot-variability manipulations assess how broadly models generalize the pattern. Removing these related constructions further tests their contribution to unseen AANN acceptability.