What quantization actually costs a small support agent (and the surprise that cost more)
We quantized a 9B support agent to GGUF at q8, q5, and q4 and measured quality, speed, and memory on a 16GB consumer GPU. q8 was free, q4 was not, and the serving harness moved the score more than the quantization did.
We took our distilled 9-billion-parameter support agent, quantized it to GGUF at q8_0, q5_k_m, and q4_k_m, and measured quality, tokens per second, and VRAM on a single 16GB consumer GPU. q8 was effectively lossless: it matched the 16-bit model exactly on the hard test set, ran about 4x faster, and actually fit the card (the 16-bit model does not, it spills into system RAM and crawls). q5 cost a little quality, q4 cost a lot and started missing escalations. The real surprise: changing the serving stack and chat template moved the score more than the quantization level did, a 4-bit model in one harness outscored a 16-bit model in another. The lesson: for a small agent, pick q8 unless size is everything, and hold your eval harness fixed or you will blame the quantization for losses the harness caused.
Distillation gives you a small model. Quantization is what makes it cheap to run. We had just distilled a multi-turn, tool-using support agent into a 9B model, so the obvious next question was: how much quality do we lose by shrinking it to 4-bit, and is it worth it?
We measured it, on one 16GB consumer GPU, and the answer had a twist we did not expect.
The setup
We took the finished 9B agent (trained in bf16) and exported it to GGUF, the format llama.cpp and Ollama use, at three quantization levels:
- q8_0 (8-bit), q5_k_m (~5-bit), q4_k_m (~4-bit).
We also kept the bf16 (16-bit, full precision) GGUF as the reference. Then we scored all four through the same runtime (Ollama) on the same held-out support conversations from two companies the model never trained on, measuring action accuracy, escalation recall, tokens per second, file size, and VRAM. For background on the format itself, see running distilled models locally; for how quantization differs from distillation as a compression strategy, see distillation vs quantization vs pruning.
What quantization costs
Here is the curve on the two held-out companies (action accuracy is the share of agent turns where it chose the right action; recall is how many of the must-escalate turns it caught):
| metric | bf16 | q8_0 | q5_k_m | q4_k_m |
|---|---|---|---|---|
| action accuracy (company A) | 70.0 | 74.0 | 65.0 | 60.0 |
| action accuracy (company B) | 62.0 | 62.0 | 58.0 | 46.0 |
| escalation recall (company B) | 46 | 62 | 62 | 39 |
| tokens/sec | 10 | 44 | 61 | 68 |
| GGUF size | 18 GB | 9.2 GB | 6.2 GB | 5.4 GB |
| VRAM on GPU | 14.6 GB + RAM | 8.9 GB | 6.2 GB | 5.5 GB |
Three things fall out of this.
q8 is effectively free. It matches bf16 (62.0 vs 62.0 on the hard company; the 4-point gap on the easy one is within noise on a 100-turn sample). And it does that at roughly 4x the speed of bf16 while using 9 GB instead of needing the full 16-bit weights. If you are running a small model and want to lose nothing, q8 is the default.
q4 is not free for this agent. Action accuracy drops from 62 to 46 on the hard company, and escalation recall falls to 39, meaning a quantized-too-far agent starts missing the tickets that need a human again. Note that JSON validity stayed high throughout, so q4 is not breaking the output format, it is making worse decisions. q5 sits in between: a few points of quality for a meaningful size cut.
The full-precision model does not even fit. bf16 at 9B is 18 GB, larger than the 16GB card, so it offloads part of itself into system RAM and runs at about 10 tokens per second, roughly a quarter of q8's speed, for zero quality gain over q8. That is the entire practical argument for quantization in one row: the un-quantized model is both slower and unable to fit, and it buys you nothing.
Q&A: So which quant should I use?
For a small structured agent like this, q8 if you can spare the space, q5 if you need it smaller, and q4 only when size is the hard constraint and you have verified the quality is still acceptable for your task. The headline numbers (size, speed) make q4 tempting, but for anything where a wrong decision has a cost, test the actual task before trusting it. This is exactly the distilled-models-need-receipts point: the leaderboard-style "it still works" is not the same as "it still escalates the security ticket."
The surprise: the harness moved the score more than the quantization
Here is the part we did not see coming. We had an earlier quality number for this model from a different evaluation, run through Hugging Face Transformers with the model in 4-bit, using the model's own built-in chat template. That harness scored the hard company at 78.7. Every Ollama number above, including the full 16-bit bf16, landed around 62.
Read that again: a 4-bit model in one harness scored higher than the 16-bit model in another. If precision were the thing driving the score, the 16-bit model would win. It did not. So the roughly 16-point gap was not quantization at all. It was the harness: the difference between the two chat templates (the model's native one versus a hand-written one in the Ollama config) and the decoding settings (pure greedy versus a runtime's default sampling). Same weights, different plumbing, different answers.
The practical warning is bigger than this one model. People obsess over which quant level to pick, but switching serving stacks, or getting the chat template slightly wrong, can cost you more quality than any quant choice would, and you will not see it in a size or speed number. If you compare a q4 served one way against a q8 served another way, you are not measuring quantization, you are measuring your own inconsistent setup. Hold the harness fixed, vary one thing at a time.
How we measured it (and the limits)
Everything above is our own measurement on synthetic, held-out support conversations, scored at every agent turn. What it does not prove:
- The companies and the gold labels are model-generated, so this is distribution-transfer, not a human-rated production test.
- Escalation recall is computed over a small number of must-escalate turns per company, so treat the recall column as directional and the action-accuracy column as the stable signal.
- We attribute the harness gap to template plus decoding, but we did not isolate which of the two dominates. The clean follow-up is a small ablation: fix the template, then fix the sampling, and re-measure to split the credit.
The takeaway
For a small, locally-run agent, quantization is mostly a gift: q8 gave us the full model's quality at a quarter of the memory and four times the speed, and it is the difference between fitting a 16GB card and not. The quality only starts to erode when you push to q4, and even then it is the decisions that degrade, not the format.
But the sharper lesson is about measurement. The thing that moved our quality number most was not the bits per weight, it was how the model was served and prompted. Quantization is the easy variable to name, so it gets the blame. Before you attribute a quality drop to "the quant," make sure you did not change the harness underneath it.
The AI Distillery is the research hub for model distillation.
Explore the knowledge base →