Distillation vs. continual learning: two bets against the monolith
Distillation and continual learning are different answers to the same problem: the giant, static, one-size-fits-all model. One compresses capability into a small specialist you run today; the other, the bet behind Sara Hooker's Adaption Labs, builds models that keep adapting on the fly.
Distillation and continual learning are two bets against the same target: the giant, static, one-size-fits-all model. Distillation compresses a large teacher's capability into a small, specialized student that you train once and run cheaply, even on a 16GB consumer GPU. Continual learning, the bet behind Sara Hooker's $50M-funded Adaption Labs, aims for models that keep adapting on the fly without expensive retraining. They are less rivals than different answers to 'capability should fit you, not you fitting the model': distillation is shippable today and hands you a fixed specialist, while continual learning is the earlier, higher-variance frontier promising a model that never stops adjusting. The throughline is that the scale-is-everything era is cracking, and the future looks like the right small model for the job, not one monolith for everyone.
For a few years the recipe for better AI was simple: make it bigger. More parameters, more data, more compute. That consensus is now visibly cracking, and two different camps are proposing what comes next. One is distillation, the thing this site is built around. The other is continual learning, the bet behind a newly-funded lab worth paying attention to. They are easy to confuse because they share an enemy, so it is worth laying out exactly how they differ and where each one wins.
The monolith they are both fighting
Both approaches react to the same thing: a single, enormous, static model that everyone queries the same way. That model is expensive to run, slow to change (you retrain or heavily prompt-engineer to adapt it), and it is tuned to the average case, which quietly erases the exceptional ones. If your use case is a niche, a specific company's support policy, a regional dialect, a narrow technical domain, the giant generalist is both overkill and underfit at the same time.
Sara Hooker, former VP of Research at Cohere, has become one of the most prominent voices arguing the scaling era has peaked. In February 2026 her new startup, Adaption Labs, raised a $50M seed round (Emergence Capital, Mozilla Ventures, Fifty Years) on the thesis that "the days of monolithic AI are over." That is the same premise behind distillation. The two camps just disagree on the fix.
Bet 1: Distillation, compress the capability into a specialist
Distillation takes a large, capable teacher model and trains a small student to reproduce its useful behavior on a task. You generate data from the teacher, fine-tune the student on it, and you are left with a compact model that does one job well and runs almost anywhere. (For the mechanics, see what is model distillation, and for how it compares to other shrinking methods, distillation vs quantization vs pruning.)
The classic proof point is DistilBERT (Sanh et al., 2019), which kept about 97% of BERT's language understanding at 40% of the size and 60% faster inference. The modern version is even more striking: we recently distilled a multi-turn customer-support agent into a 9-billion-parameter model that runs on a single 16GB consumer GPU (see distilling a customer-support agent), for a few dollars of teacher API.
The shape of the bet:
- When adaptation happens: once, at training time. You bake the specialization in, then freeze it.
- What you get: a fixed, predictable, auditable artifact you fully own and can run offline.
- The cost: it is static. When the policy or the world changes, you regenerate data and distill again. It does not learn from yesterday's conversations on its own.
Distillation is mature. You can do it today, on hardware you already own, with a known recipe.
Bet 2: Continual learning, a model that keeps adapting
Continual learning aims at the opposite property: a model that keeps adjusting after deployment, learning from new data and from how it is used, without an expensive retraining cycle and without forcing the user to become, in Hooker's framing, an "elevated prompt engineer" compensating for the model's limits.
Adaption Labs organizes this around three pillars: adaptive data (systems that generate and shape the data they need on the fly rather than depending on one big static dataset), adaptive intelligence (spending more or less compute based on how hard the problem is), and adaptive interfaces (learning from user interaction). In May 2026 they launched AutoScientist, a system that runs the full fine-tuning loop end-to-end, co-optimizing the training data and the model configuration together.
The shape of the bet:
- When adaptation happens: continuously, after deployment.
- What you get: a model that tracks your specific use over time rather than a fixed snapshot.
- The cost and risk: this is the harder, earlier frontier. Continual learning has to fight catastrophic forgetting, stability, and evaluation (how do you certify a model that changes under you?). Much of the promise is still being proven in public.
How they compare
| Distillation | Continual learning | |
|---|---|---|
| Adapts | once, at training time | continuously, after deployment |
| Result | a fixed, ownable specialist | a model that keeps changing |
| Runs where | locally, on consumer hardware | depends on the system; often a platform |
| Maturity | proven, shippable today | early, frontier, largely unproven at scale |
| Main weakness | static; re-distill to update | stability, forgetting, hard to certify |
| Control | you own the artifact | the system manages adaptation |
Q&A: Are they competitors or complementary?
Mostly complementary; they operate at different layers. Nothing stops you from distilling a small specialist and then giving it a continual-learning loop on top, in fact a small model is a far cheaper thing to keep adapting than a frontier monolith, so distillation and continual learning may end up as partners rather than rivals. The interesting middle ground already exists in the literature as on-policy distillation, where the student keeps learning from its own outputs with teacher feedback, a step toward adaptation that still borrows the teacher's knowledge.
Q&A: Which should I use today?
If you need a specialized model in production now, distillation is the answer that ships. It is well understood, runs on a 16GB card, and produces an artifact you can test, version, and trust. Continual learning is the more exciting long-term bet, and it is worth tracking closely, especially as a well-funded lab like Adaption Labs pushes it, but as of mid-2026 it is the higher-variance research frontier, not the thing you reach for to solve a concrete problem this quarter.
The shared throughline
Strip away the mechanism and the two camps are saying the same thing: the era of one giant model for everyone is ending, and capability is migrating toward the right small model for the job. Distillation gets you there by compressing a frontier teacher into something you own and run yourself. Continual learning gets you there by building a model that never stops fitting itself to you.
When a researcher of Hooker's standing raises $50M to bet against scaling, it is a useful signal that the niche this site has been mapping, putting frontier capability into small, specialized, individually-owned models, is not a backwater. It is where a meaningful slice of the field is heading. We just happen to think the version you can run on the GPU in the other room is the most interesting one.
The AI Distillery is the research hub for model distillation.
Explore the knowledge base →