specimen 07 / distilled
Distilling intelligence
into models you can hold.
The knowledge centre and research hub for model distillation: turning frontier models into small, fast, open ones that run on your own hardware. Where the applied frontier is documented and pushed forward, from first principles to the methods being invented now.
teacher → reasoning trace → student → quantize → run it local
What is model distillation?
A large teacher model knows far more than its size lets most people use. Knowledge distillation transfers that understanding into a smaller student, not by copying weights, but by learning from the teacher’s soft predictions, its reasoning traces, and the synthetic data it generates.
The result is a model a fraction of the size that keeps much of the capability, small enough to run on a laptop, a phone, or a single GPU in your closet. Distillation is how frontier intelligence becomes something you own.
@@#@@#@ #@@@@#@ @#@@#@@ @@#@@@# #@@#@@@
·:+:· :+*+: ·:+:·
Intelligence that fits where you need it.
Faster
A distilled student answers in a fraction of the time and cost of its teacher: real-time on modest hardware.
Smaller
From hundreds of billions of parameters to a handful, small enough for a laptop, edge device, or phone.
Yours
Run it offline, on-prem, private. No tokens metered, no data leaving the building, no rate limits.
Specialized
Distill only the capability you need. A focused student can rival a giant on its narrow domain.
Learn the craft, end to end.
A structured path from first principles to the techniques at the edge of the research.
What is model distillation?
A plain-language primer on knowledge distillation — how a small student model learns to think like a giant teacher, and why it's the key to running AI on your own hardware.
How distillation works: the three kinds of knowledge
Response, feature, and relation-based distillation — plus self, online, and offline variants. The conceptual map of how knowledge actually moves from teacher to student.
The distiller's toolkit
The frameworks people actually use to distill models in 2026 — from Hugging Face TRL and Arcee DistillKit to synthetic-data pipelines and managed cloud services.
Reasoning distillation: teaching small models to think
How chain-of-thought traces turned distillation from a compression trick into a way to transfer reasoning itself — the DeepSeek-R1 recipe and why it changed the field.
From the research hub.
Distillation vs. continual learning: two bets against the monolith
Distillation and continual learning are different answers to the same problem: the giant, static, one-size-fits-all model. One compresses capability into a small specialist you run today; the other, the bet behind Sara Hooker's Adaption Labs, builds models that keep adapting on the fly.
5 min read →What quantization actually costs a small support agent (and the surprise that cost more)
We quantized a 9B support agent to GGUF at q8, q5, and q4 and measured quality, speed, and memory on a 16GB consumer GPU. q8 was free, q4 was not, and the serving harness moved the score more than the quantization did.
6 min read →When distillation beats fine-tuning — and when it doesn't
A new June 2026 paper studies knowledge distillation during post-training and lands on the practical rule: distillation helps most when data is scarce or the teacher knows something the dataset does not.
6 min read →
Where the cutting edge of distillation is documented.
Distillation is moving fast, and we distill models ourselves. This is the serious place of record for the applied frontier: the techniques, the real numbers, and original research, published as the field is built.