Unsloth Dynamic: not "how many bits" but "how many bits where"
Every format so far picks one bit-width and applies it nearly uniformly. That leaves an enormous amount on the table, because sensitivity to precision loss varies by orders of magnitude between tensor types — and the pattern differs for every architecture.
The DeepSeek-R1 demonstration
The result that made the technique famous: DeepSeek-R1, 671B parameters, 720 GB → 131 GB, an 80% reduction, still functional. Unsloth reported the 1.58-bit build fitting in 160 GB of VRAM (2×H100 80GB) at roughly 140 tokens/sec throughput.
The architectural reasoning behind the allocation:
| Component | Share | Bits | Why |
|---|---|---|---|
| First 3 dense layers | ~0.5% | 4–6 | Establish the representation everything downstream depends on |
| MLA attention (61 layers) | ~5% | 4–6 | Compressed latent KV — errors propagate through every token interaction |
| Shared experts | ~1.5% | 6 | Active on every token, so error is never averaged away |
down_proj | large | higher | Sits after the SwiGLU gate, inherits the largest activation magnitudes |
| MoE expert layers (58×) | ~88% | 1.58 | Massively redundant; each expert fires on a small token fraction |
The control experiment is the part that matters. On Unsloth's pass@3 Flappy Bird code-generation test:
| Build | Size | Score |
|---|---|---|
| Naive — every layer 1.58-bit | — | 0% |
| Naive low-bit build | 175 GB | 61.7% |
| Dynamic 1.58-bit | 131 GB | 69.2% |
| Dynamic 2.22-bit | 183 GB | 91.7% |
Read the first two rows again. Uniform 1.58-bit doesn't degrade the model — it annihilates it, producing infinite repetition loops and gibberish. And the 175 GB naive build is both bigger and worse than the 131 GB dynamic one. Bit allocation isn't a tuning detail; it dominates the size–quality frontier.
What Dynamic 2.0 changed
The original technique was hand-derived and worked specifically on mixture-of-experts models, where "88% of weights are redundant experts" makes the win obvious. Dynamic 2.0 generalised it:
- Every tensor is a decision. Rather than promoting a hand-picked list, the quantization type is chosen per-tensor, and the combination differs per model — Unsloth notes the layers quantized in Gemma 3 differ significantly from those in Llama 4.
- It works on dense models, not just MoE.
- A purpose-built calibration set of 1.5M+ tokens of curated chat and code data.
- Evaluation by KL divergence, not perplexity.
- Extra formats (
Q4_NL,Q5_1,Q5_0,Q4_1,Q4_0) for Apple Silicon and ARM, where flat block layouts map better to the available SIMD instructions.
Implementation note: the per-tensor assignment is expressed through llama.cpp's quantization machinery, not a separate inference-time mechanism. Decisions are baked into the GGUF at export; llama.cpp then loads it as an ordinary file. Nothing is "dynamic" at runtime — the intelligence is all in choosing the layout.
Why KL divergence, not perplexity
Perplexity averages log-likelihood over a corpus, and errors can cancel: a quantized model that becomes more confident on some tokens and less on others can post an unchanged perplexity while behaving quite differently.
The paper Accuracy is Not All You Need introduced flips — answers changing from correct to incorrect or vice-versa. A quantized model can hold its MMLU score steady while flipping a large fraction of individual answers in both directions. The score is preserved; the model is not.
KL divergence measures what we actually care about: are you still the same function?
D_KL(P_full ‖ P_quant) = Σ_t P_full(t) · log( P_full(t) / P_quant(t) ) // averaged over every token position in a held-out corpus // 0 = indistinguishable from the original
| Quant | Baseline | GB | Dynamic 2.0 | GB | Δ |
|---|
Two things are visible. First, the frontier is brutally non-linear: between 6 and 10 GB, MMLU climbs 27 points; between 15 and 27 GB it climbs 0.1 and then declines. Almost all the value is in the first few bits.
Second, the marked point: Google shipped a quantization-aware trained Gemma 3 27B at 17.2 GB scoring 70.64. Unsloth's post-training Q4_K_XL is 15.64 GB and scores 71.47 — smaller and higher. Good bit allocation applied after the fact competed with retraining the model.
Build your own allocation
A simplified sensitivity model of an MoE transformer. Assign bits per component and watch size and estimated divergence move. Try "uniform 1.58" versus "dynamic".