Chapter 07

Which one should you actually download?

The frontier from chapter 4 compresses into a fairly simple rule. Assume you want the best model that fits your memory.

Available memory	Choose	Why
Fits at Q8_0	`Q6_K`	Effectively indistinguishable from BF16; Q8_0 rarely earns its extra 2 bpw
Fits at ~4.5 bpw	`Q4_K_M` / `Q4_K_XL`	The sweet spot. KLD ~0.02. Where almost everyone should sit
Slightly short at 4-bit	`IQ4_XS` or `Q3_K_XL`	IQ4_XS is ~4.25 bpw with codebook efficiency; Q3_K_XL is a real but acceptable step down
Only 2–3 bpw fits	`IQ2_M` / `Q2_K_XL`	Meaningful degradation, but a bigger model at 2-bit usually beats a smaller model at 4-bit
Desperate	`IQ1_M` over `IQ1_S`	The 0.19 bpw from IQ1_S to IQ1_M buys a large quality jump — see the KLD chart
GPU serving, high batch	AWQ or FP8	GGUF is optimised for single-stream and CPU; GPU-native formats batch better
Fine-tuning	NF4 + QLoRA	bitsandbytes format, integrates with the PEFT/Unsloth training path

Rules of thumb worth internalising

A bigger model at lower precision usually beats a smaller model at higher precision — down to about 3 bits. Below that the relationship inverts.
The size gap between "same file size" quants is not the same as the quality gap. Two 8 GB files can differ by 5 MMLU points based on allocation and calibration alone.
Judge by KL divergence, not perplexity — and be suspicious of any perplexity number measured on data resembling the calibration corpus.
Below 3 bits, always use an imatrix-calibrated build. The IQ formats aren't merely better there; they're the only ones that function.
Check whether the quantizer fixed the model's bugs. Chat templates, RoPE scaling configs, BOS handling and epsilon values are routinely wrong in original releases. A quant built on a broken config inherits the break — and it looks exactly like quantization damage.
Don't quantize the embedding and output layers aggressively. They're a small fraction of parameters and a large fraction of sensitivity. Almost every good build leaves them at 6–8 bits.

Reproduce any of this yourself

# 1 — convert to GGUF at full precision
python convert_hf_to_gguf.py ./model \
    --outfile model-f16.gguf --outtype f16

# 2 — collect the importance matrix from a calibration corpus
./llama-imatrix -m model-f16.gguf -f calibration.txt \
    -o imatrix.dat --chunks 200

# 3 — quantize, with per-tensor overrides (the "dynamic" part)
./llama-quantize --imatrix imatrix.dat \
    --tensor-type "attn_v=q6_K" \
    --tensor-type "ffn_down=q6_K" \
    --token-embedding-type q6_K \
    --output-tensor-type q6_K \
    model-f16.gguf model-IQ2_XXS.gguf IQ2_XXS

# 4 — measure what you did, against the f16 original
./llama-perplexity -m model-IQ2_XXS.gguf -f test.txt \
    --kl-divergence-base f16-logits.dat --kl-divergence

Step 4 is the one people skip and shouldn't. It's the only way to know whether step 3 helped. Use a held-out corpus that does not resemble your calibration data, or you're grading your own homework.

The whole series in one paragraph

Summary. Quantization works because trained weights occupy a tiny, predictable slice of the range their format can represent, and because inference is starved for memory bandwidth rather than arithmetic. You recover the wasted bits by factoring out a shared scale per small block, shaping the grid to match the weight distribution, and — the part that matters most below 4 bits — measuring which weights actually influence the output and spending your bit budget there. At that point "how many bits" stops being one number and becomes a per-tensor allocation problem, which is exactly what Unsloth's dynamic quants solve, and why a 131 GB build of a 720 GB model can still write working code while a naive 175 GB build cannot.

The seven things to remember

Decoding is memory-bound. tokens/s ≈ bandwidth ÷ model bytes.
Quantization = store integers plus a shared scale. Everything else is detail.
One outlier ruins a per-tensor scale. Block-wise scaling costs ~0.5 bpw and fixes it.
Grid shape matters: NF4 puts levels where the probability mass is.
k-quants quantize the scales too, in a two-level hierarchy.
Below 4 bits, importance-weighted objectives and codebooks are mandatory, not optional.
Bit allocation across tensors dominates the size–quality frontier.

Where to go next

Unsloth — Dynamic 2.0 GGUFsMethodology, the Gemma 3 MMLU and KLD tables, QAT comparison Unsloth — DeepSeek-R1 Dynamic 1.58-bit720GB → 131GB, layer allocation, the Flappy Bird benchmark llama.cpp — Tensor Encoding SchemesExact bpw and block structure for every GGUF type QLoRA (Dettmers et al.)NF4, double quantization, paged optimisers Accuracy is Not All You NeedFlips, and why KL divergence is the right metric BitNet b1.58What happens when you never leave ternary