Research program · ANUPNET
Language models that keep learning — without forgetting.
A small language model that learns from a single unlabeled stream of new domains, stores no old data, replays nothing, and is never told when the domain changes — while keeping most of what it learned before.
88%
less catastrophic forgetting than naive fine-tuning (2.74 → 0.33 nats, 3 seeds, 30M model)
0
stored data, replay buffers, or task labels used by the mechanism
4 / 4
domain boundaries detected at the exact step in an unlabeled stream, 0 false alarms, 3 seeds
110M
parameter model pretrained on 1B tokens as the base for the next stage
The problem
Today's models are snapshots.
Every large language model is frozen the day its training ends. Teach it a new domain and it overwrites the old one — catastrophic forgetting. The standard fixes either store old data and replay it, or need a label telling the model which task it is on. Humans do neither: skills, gist and connections stay while details fade.
ANUPNET is built under three rules we do not break: no stored old data, no task labels, and no absolute numbers in decisions — every threshold is a ratio to the model's own running statistics.
How it works
A hall, rooms, dreams and sleep.
1
Hall and rooms
The trunk ("hall") is frozen after the first domain. Every FFN and attention block gets small gated side modules ("rooms") — one per domain, opened only when its domain is recognised.
2
Dreams as contrast
A gate cannot learn old-vs-new without seeing old inputs. Instead of a data buffer, the model generates its own text ("dreams") and uses it only to train the gates — never the weights.
3
Sleep
When a room closes, it learns to stay silent on dreams, and older rooms' keys are refreshed so they still recognise their own domain as the input distribution drifts.
4
Novelty and familiarity
A loss jump above the model's own noise band opens a new room before the step. A dormant room whose gate wakes up on the input is reopened instead — so revisits go back to the right memory.
Results · 30M model, wiki → code → stories
Measured, seeded, with controls.
| Method | Stored data | Avg forgetting ↓ |
|---|---|---|
| Capacity-matched control (same extra params, no mechanism) | – | 3.70 |
| Naive fine-tuning | – | 2.74 |
| Freeze trunk only | – | 2.58 |
| Dream gate (FFN rooms) | – | 0.87 ± 0.04 |
| ANUPNET (+ sleep + attention rooms) | – | 0.33 ± 0.05 |
| Replay 30% (stores old data — reference) | yes | 0.05 |
Forgetting = rise in validation loss (nats) on old domains after training on new ones; 800 steps per domain; 3 seeds for the two headline rows. What we are not claiming: ANUPNET does not beat replay when storing data is allowed; new-domain acquisition costs ~0.45 nats; results are at 30M parameters and 3 domains. All negative results are in the report.
Roadmap
From 30M to 125M.
Done
A · Pretrain
110M params, 1.05B tokens (FineWeb-Edu + Cosmopedia), val loss 3.30, 6 GPU-hours on one NVIDIA RTX Pro 6000.
In progress
B · Instruction tune
Q&A format on SmolTalk so the model can answer questions.
Next
C · Search tool
Facts live outside the weights; the weights keep skills and gist.
Q4 2026
D · Continual stream
Rooms, dreams and sleep at 125M across 10–30 domains, 3 seeds, plus a compute-cost meter vs replay and retraining.
Code, configs and the full experiment log (including failures) will be released with report v1. Compute partners and reviewers welcome — admin@lexinexiai.com.