Infinite-Parameter LLMs Generate Weights from Live Data — SkimNews
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- The Infinite-Parameter LLM paper proposes a compact hypernetwork that turns runtime data into a low-rank modulation of a shared base network, so feed-forward weights are generated from live interaction rather than stored in a fixed parameter bank.
- The architecture carries a Bayesian belief over the generator's latent code and updates it online, re-deriving the effective weight from that evolving belief as a session proceeds instead of fixing the weight after one read of the context.
- Authors of the paper, submitted to arXiv on 16 Sep 2026, claim the model maintains a fixed stored footprint while the weights it can compile are "effectively infinite."
- The approach is inspired by Mixture-of-Experts (MoE) architectures, which activate only a fraction of an enormous stored parameter bank per token during static pretraining — the new design replaces static storage with runtime generation.
- The paper argues that carrying runtime-supplied knowledge and behavior in weights rather than the prompt is amortized in compute, frees the context window, persists across turns, and can generalize better than in-context use.
- The authors specify an evaluation protocol explicitly designed to test the architecture against in-context learning and retrieval baselines.
Why it matters: Today, a deployed LLM can only treat user-supplied facts and corrections as transient prompt content that disappears when the request ends. This paper proposes an architecture that writes that runtime knowledge into the model's effective weights, which — if the evaluation bears it out — would give a fixed-footprint model persistent, generalizable memory of live interaction without enlarging storage.
Ask SkimNews
