One-sentence Summary

Researchers from Nace AI and Purdue University establish scaling laws for hypernetwork-based train-time knowledge injection, wherein a hypernetwork generates LoRA adapters to enable factual reasoning in large language models, revealing power-law scaling with hypernetwork depth, width, and target model size and reliable out-of-distribution generalization that outperforms LoRA fine-tuning, supported by their large-scale MegaWikiQA dataset.

Key Contributions

• MegaWikiQA, a large-scale dataset of tens of millions of deterministic multi-hop question-answer examples across 39 knowledge domains, is introduced for controlled scaling experiments and out-of-distribution evaluation in knowledge injection.
• The first systematic scaling laws for hypernetwork architectures in train-time knowledge injection are presented, characterizing power-law scaling of loss, reasoning accuracy, and out-of-distribution performance with hypernetwork width, depth, target model size, and number of injected facts.
• Hypernetwork-based injection yields steeper scaling exponents and more reliable out-of-distribution generalization to unseen entities and relations than LoRA fine-tuning or full fine-tuning under matched capacity, establishing it as a principled and scalable method for factual reasoning in large language models.

Introduction

Large language models must reliably internalize and generalize domain-specific factual knowledge for sensitive applications like medicine, law, and finance, where full fine-tuning is costly and even parameter-efficient methods (e.g., LoRA) suffer from catastrophic forgetting and poor out-of-distribution generalization. Hypernetworks offer an alternative by generating LoRA-style weight adaptations conditioned on a corpus of facts, without modifying the base model parameters, but their scaling behavior remained unknown. The authors present the first systematic empirical scaling laws for hypernetwork-based knowledge injection, analyzing how performance varies with hypernetwork width, depth, target model size, and the number of injected facts, using a new large-scale dataset called MegaWikiQA. They find that hypernetwork adaptation exhibits smooth predictable scaling and substantially better out-of-distribution generalization than standard finetuning approaches as model size grows.

Dataset

The authors construct MegaWikiQA, a large-scale dataset for studying knowledge injection, entirely from the Wikidata5M knowledge graph.

• Sources and foundation: Wikidata5M contains ~4.6M entities, 822 relations, and over 22 million triplets. Its graph structure and unambiguous triplets support deterministic generation of both single-hop and multi-hop question-answer pairs.
• Question generation: k-hop walks (k = 1–4) are randomly sampled across the graph. A grammar-based pipeline converts each walk into a natural-language question and answer using manually curated templates for each relation. Only one-to-one and many-to-one relations are retained to guarantee deterministic, unambiguous answers.
• Filtering and balancing: Every example is assigned to one of 39 knowledge domains through a two-stage domain classifier. Examples with uncertain domain labels are removed. The initial pool of roughly 10 million examples per hop count is then balanced across domains and hop counts, yielding a final training set of 1.25 million samples.
• Fact injection protocol during training: Each training sample consists of an injected fact set, a query, and a ground-truth answer. The fact set includes one relevant fact (drawn from the k-hop walk that generated the query) and N-1 distractor facts sampled uniformly at random from the full fact collection. Models must reason over this set to answer.
• Evaluation splits:
  • In-distribution (ID): 10,000 examples, stratified by domain and hop count, with no triplet overlap with the training set.
  • Out-of-distribution (OOD): 10,000 examples drawn exclusively from three held-out domains (philosophy, linguistics, civil engineering). This set includes both original grammar-based questions and GPT-4.1 rephrased versions to probe generalization beyond surface form.
  • MCQ split: constructed from the same OOD examples, where each question offers four choices (one correct plus three random distractors).

Method

The authors study the problem of knowledge injection, where a large corpus of facts $\Omega$Ω is injected into a frozen language model $\mathcal{M}_\theta$Mθ​ with fixed parameters $\theta$θ. The core objective is to produce an answer $a$a consistent with a relevant fact from an injected fact set $\mathcal{F}$F, without modifying the base model parameters $\theta$θ at any point. Only the hypernetwork parameters $\phi$ϕ are updated during training, which helps mitigate the high cost of fine-tuning and facilitates out-of-distribution generalization.

As shown in the figure below, the proposed framework utilizes a transformer-based hypernetwork $g_\phi$gϕ​ to generate contextual weight adaptations for the target model. The hypernetwork maps an input fact set $\mathcal{F}$F to a collection of LoRA-style weight adaptations, which are applied to the frozen target model during the forward pass. The target model remains fully frozen throughout both training and inference.

The hypernetwork is initialized entirely from random weights to avoid conflating the effect of pretraining with architectural capacity. It employs a LoRA rank of $r = 4$r=4 and a scaling factor of $\alpha = 8$α=8. The authors systematically scale the transformer hypernetwork along three independent axes to characterize scaling laws: depth (varying the number of transformer layers $L_{\text{HN}}$LHN​), width (varying the hidden dimension $d_{\text{model}}$dmodel​), and fact count (varying the number of injected facts $N$N per example).

During training, each example consists of a natural language query $q$q and a set of $N$N injected facts $\mathcal{F} = \{f_1, \dots, f_N\} \subset \Omega$F={f1​,…,fN​}⊂Ω. Exactly one fact in $\mathcal{F}$F is relevant to answering $q$q, randomly selected from the k-hop sequence of facts used to generate the question-answer pair. The remaining $N - 1$N−1 facts are negative facts sampled uniformly at random from $\Omega$Ω. This setup encourages the hypernetwork to identify and utilize the relevant fact from a noisy input context, translating it into weight adaptations that steer the frozen target model toward the correct answer. The authors fix $N = 4$N=4 facts per example across most experiments, studying the effect of varying $N$N in dedicated scaling experiments.

Experiment

Using a comprehensive suite of scaling experiments on the MegaWikiQA dataset with frozen Qwen2.5 language models, the paper examine how hypernetwork width, depth, target model size, and injected fact count influence in-distribution loss and three forms of out-of-distribution generalization. All dimensions show smooth power-law improvements, but scaling the target model yields the steepest gains per unit compute, while linguistic robustness to rephrased questions remains the most difficult to improve across any single axis. When comparing knowledge injection via hypernetworks versus direct finetuning (LoRA and full), the hypernetwork’s advantage on out-of-distribution tests grows monotonically with target model size, making it increasingly favorable for reliable generalization at scale.

Target model size is the most effective scaling axis, yielding steeper loss reductions than hypernetwork width, depth, or fact count. In-distribution scaling favors fine-tuning methods slightly, but the hypernetwork scales better on all out-of-distribution metrics, with its relative advantage increasing at larger model sizes. OOD rephrased evaluation consistently shows the flattest scaling, indicating that robustness to linguistic variation is the hardest to improve through scaling. Target model size scaling produces the steepest exponents across all metrics, with hypernetwork target scaling reaching −0.226 on in-distribution validation, compared to −0.096 for width, −0.088 for depth, and −0.080 for fact count. OOD rephrased metrics have the shallowest exponents across every axis (e.g., −0.036 for HN width, −0.028 for fact count), making linguistic robustness the least responsive to scaling. On in-distribution validation, LoRA fine-tuning (−0.250) and full fine-tuning (−0.249) scale slightly more favorably than the hypernetwork (−0.226). On all OOD splits, the hypernetwork exhibits steeper scaling than fine-tuning, notably on OOD rephrased (−0.107 vs. −0.083 LoRA, −0.069 full FT) and OOD MCQ (−0.171 vs. −0.119 and −0.101). The OOD advantage of the hypernetwork widens as target model size grows, so the hypernetwork paradigm becomes increasingly beneficial at larger scales.

This study scales target model size, hypernetwork width and depth, and knowledge facts, comparing a hypernetwork-based approach to LoRA and full fine-tuning on in-distribution and out-of-distribution generalization. Target model size emerges as the most effective axis for loss reduction across all methods. While fine-tuning scales slightly better in-distribution, the hypernetwork shows steeper improvement on all out-of-distribution metrics, with its advantage widening at larger model sizes; linguistic robustness to rephrasing remains the hardest capability to improve through any scaling dimension.