Injecting factual knowledge into LLMs at scale is still an unanswered engineering problem: fine-tuning every model or keeping enormous retrieval stores can be costly or brittle. This paper shows that using a hypernetwork to generate a fixed LoRA adapter at train time cleanly decouples injection capacity from the base model and enables a controlled study of how injection performance scales with architecture and model size.
Key Findings
- Empirical power-law scaling: loss and reasoning accuracy follow broadly predictive power-law behavior when varying hypernetwork depth, width, and target model size — so you can predict returns from scaling decisions instead of relying on ad-hoc tuning.
- Better OOD generalization at scale: as hypernetworks grow, they generalize more reliably to out-of-distribution queries, indicating that train-time hypernetwork injection can transfer beyond the exact facts seen during training.
- Competitive against other train-time methods: hypernetwork-generated LoRA adapters exhibit steeper scaling exponents in OOD evaluations compared to conventional LoRA fine-tuning and full fine-tuning, implying more favorable returns with scale.
- Large evaluation corpus: the authors construct MegaWikiQA (tens of millions of multi-hop QA examples across 39 domains) to benchmark factual reasoning and out-of-distribution performance — so claims are validated on a wide, multi-domain dataset.
Who it's for & tradeoffs
Great fit if you need to inject large corpora of static factual knowledge into many models or model sizes and want predictable scaling behavior and improved OOD robustness. The approach reduces per-model manual fine-tuning by generating adapters centrally.
Look elsewhere if you require instant, per-query updates to facts (online updates are not the focus here), or if minimal infrastructure and very low-latency per-request edits are your primary constraints. Hypernetworks add training complexity and an extra artifact (generated adapters) to manage, and they still require substantial compute and data to reach the best scaling regimes.
Methods and dataset
The method trains a hypernetwork to map sets of facts (from a large corpus) to weights of a fixed LoRA adapter; that adapter is then inserted into a target LLM to enable answering fact-based questions. The paper evaluates depth/width/target-size axes and measures loss, reasoning accuracy, and OOD generalization. MegaWikiQA — a constructed dataset containing tens of millions of multi-hop QA examples across 39 domains derived from Wikidata5M — is used for large-scale evaluation and to measure out-of-distribution transfer.