Sixteen bits go in.
One and a half come out.
The model still talks.

A 671-billion-parameter model that needs 720 GB of memory gets squeezed into 131 GB — and still writes working code. This series takes that apart: the arithmetic, the file format, the calibration statistics, and the exact instructions a GPU runs when it multiplies by a 4-bit number.

Start here: quantization isn't about compute

The intuitive story — "fewer bits means faster math" — is mostly wrong for LLM inference. Generating a token is not compute-limited. It's limited by how fast weights can be dragged out of memory.

To produce one token at batch size 1, the model must read every weight exactly once and do exactly two floating-point operations per weight. That ratio is fixed by the math of a matrix–vector product.

// arithmetic intensity of decoding one token, batch 1
FLOPs = 2 × P            // P = parameter count
Bytes = P × bits / 8

FP16 : 2P / 2P    = 1.0 FLOP per byte
INT4 : 2P / 0.5P  = 4.0 FLOP per byte

// an H100 sustains ~990 TFLOP/s on ~3.35 TB/s of HBM
// its ridge point is ~295 FLOP per byte
→ decoding sits ~300× into memory-bound territory

The GPU spends almost all of a decode step waiting on HBM. Halve the bytes per weight and you roughly double the token rate — without making the arithmetic any cheaper. The FLOPs were never the bottleneck.

There's a second win. A 70B model at BF16 is 140 GB — it doesn't fit on one GPU, so you pay for tensor-parallel communication. At 4.5 bpw it's ~39 GB and fits on a single card. You didn't just make it faster; you removed an entire class of interconnect cost. That's why quantization research is obsessed with the region between 2 and 5 bits — that's where the capacity cliffs are.

The seven chapters