Chapter 06

If you can afford gradients, you can do better than rounding

Everything so far is post-training quantization: take a finished BF16 model and compress it. Five alternatives use gradients, or avoid the problem entirely by never leaving low precision.

Method	Needs	Core idea
GPTQ	calibration data	Quantize column by column, correcting remaining weights using the Hessian
AWQ	calibration data	Rescale the ~1% of channels with the largest activations before quantizing
QAT	full training run	Put fake-quantize in the forward pass so the model learns to survive it
BitNet b1.58	pre-training from scratch	Weights are ternary throughout training — no approximation error at all
QLoRA	fine-tuning run	Freeze a 4-bit base, train small BF16 adapters on top

GPTQ — second-order error correction

Instead of rounding all weights independently, quantize them one column at a time and, after each, adjust the remaining unquantized weights to compensate for the error just introduced. The compensation direction comes from the layer's Hessian, approximated from calibration activations as H ≈ 2XXᵀ.

It's not training — no backprop through the network — but it is a proper optimisation rather than a rounding rule, and it substantially beats naive round-to-nearest at 3–4 bits.

// sketch: for each column j, left to right
q[:,j]  = quantize(W[:,j])
err     = (W[:,j] − q[:,j]) / H⁻¹[j,j]
W[:,j+1:] −= err ⊗ H⁻¹[j, j+1:]   // push the error forward

AWQ — protect the salient channels

Activation-aware Weight Quantization starts from the same insight as the importance matrix but acts differently. It identifies the ~1% of weight channels that see the largest activations and applies a per-channel scaling before quantizing, so those channels effectively get more resolution.

Because the scaling is folded into the preceding layer's output, there's zero inference-time cost. Popular in vLLM deployments where GPU-native throughput matters more than the last fraction of a bit.

QAT — train with the rounding in the loop

Insert fake-quantize operations into the forward pass so the network experiences quantization error during training and learns weights that survive it. Gradients flow through the non-differentiable rounding via the straight-through estimator: forward rounds, backward pretends it was the identity.

// straight-through estimator
forward :  ŵ = s · round(w/s)
backward:  ∂L/∂w = ∂L/∂ŵ        // pretend round() was identity

Google's Gemma 3 QAT release is exactly this. QAT is expensive — and as the chapter 4 charts showed, a well-allocated post-training quant matched and slightly beat it at a smaller file size. That's a strong argument for the "where do the bits go" framing over the "train harder" one.

BitNet b1.58 — never leave ternary

The most radical option: train from scratch with weights constrained to {−1, 0, +1}. Since the weights are ternary throughout training, there is no post-hoc approximation error at all — the network simply learns within that constraint.

Matrix multiplication reduces to additions and subtractions, which is a fundamentally different hardware story: no multipliers needed.

The catch. You must pre-train the model this way. You cannot convert an existing BF16 model into a BitNet, which is why the ecosystem is still dominated by post-training quantization of conventionally-trained models.

QLoRA — fine-tune on a frozen quantized base

The technique that made 4-bit practically important for training, not just serving. The recipe:

Freeze the base model in NF4 with double quantization. It never receives a gradient.
Attach small trainable low-rank adapters (LoRA) in BF16 alongside each frozen linear layer.
During the forward pass, dequantize each 4-bit block to BF16 on the fly, in registers, use it, discard it. Never materialise a BF16 copy of the whole model.
Gradients flow through the frozen weights to the adapters. Only adapter parameters live in the optimiser state.

y = dequant(W_NF4) · x  +  (B · A) · x
    └──── frozen, 4-bit ──┘   └─ trainable, BF16 ─┘
                                  rank r ≪ d

// memory: 4-bit base + tiny adapters + optimiser state
// for adapters only.
// A 70B full fine-tune needs ~1.2 TB. QLoRA fits under 48 GB.

Two further tricks from the QLoRA paper make it work in practice: double quantization (chapter 1) and paged optimisers, which use unified memory to page optimiser state to CPU RAM during gradient-checkpointing spikes instead of OOMing.

Why this ties back. This is the mechanism behind Unsloth's fine-tuning stack, and it's why one organisation cares about both quantization quality and training: a bad 4-bit base means every adapter trained on top inherits its damage. Quantization error and fine-tuning quality are the same problem viewed from two ends.

Where each one fits

Serving on GPU with high batch sizes → AWQ or GPTQ (or FP8 on Hopper/Blackwell). GPU-native kernels, good batching.
Serving on CPU, Apple Silicon, or a single consumer GPU → GGUF k-quants / IQ quants via llama.cpp. Best single-stream performance and the widest format range.
Fine-tuning → NF4 + QLoRA.
You control pre-training and want maximum efficiency → QAT, or BitNet if you're willing to commit to ternary from the start.