Chapter 01

Bits, the grid, and block scaling

Before you can shrink a number, you have to know where it keeps its information — and then discover that one bad weight can ruin sixteen million good ones.

What a float actually stores

A floating-point number is three fields glued together: a sign, an exponent (which sets the scale), and a mantissa (which sets the precision within that scale):

value = (−1)^sign × 2^(exponent − bias) × 1.mantissa

The exponent is why floats can represent both 10⁻⁸ and 10⁸. It gives a logarithmic spread of representable values — dense near zero, sparse far away. That property is exactly what neural network weights want, and it's the biggest clue for designing a 4-bit format.

Tap the bits below. Flipping an exponent bit doubles or halves the value; flipping a mantissa bit nudges it.

Float bit inspector

tap any bit to flip

Decoded value

—

Total bits

—

Exponent range

—

Decimal digits

—

BF16 is FP32 with the bottom 16 mantissa bits chopped off. Same exponent field, same dynamic range, a quarter of the precision — which is why FP32→BF16 never overflows, and why training frameworks adopted it over FP16.

Now the key observation: trained weights are not spread across that huge dynamic range. Within any one weight matrix they form a tight, roughly zero-centred, bell-shaped cloud. The exponent field — 8 whole bits in FP32 — is being paid for and almost entirely wasted.

The opportunity. If a whole block of weights shares roughly the same magnitude, store the exponent once for the block and give each weight a few bits of "where in the block's range am I". That single idea — factor out the shared scale — is the foundation of every scheme in this series.

The core operation

Strip away the acronyms and this is all of it. Take a group of weights. Pick a scale s. Divide, round, clamp, store the integer. Multiply back to use it.

// symmetric absmax quantization, b bits
// quantize — offline, once
qmax = 2^(b−1) − 1              // b=4 → 7
s    = max(|w|) / qmax          // one FP16 number per block
q[i] = clamp(round(w[i]/s), −qmax−1, qmax)

// dequantize — at inference, in a register
ŵ[i] = s × q[i]

Two things are stored: an array of low-bit integers and one high-precision scale. Reconstruction error is bounded by s/2 — half a grid step. Everything else in quantization research is an argument about where to put the grid steps and how many weights share a scale.

Symmetric vs. asymmetric

symmetric  : ŵ = s·q         // grid centred on zero
asymmetric : ŵ = s·q + m     // grid can slide; costs more metadata

Weight distributions are near-symmetric, so Q4_0 (symmetric) is fine. Activation distributions after a ReLU or GELU are one-sided, so activation quantization almost always goes asymmetric.

Live quantizer

4096 synthetic weights ~ N(0,1)

Bits per weight — 4

Block size (weights per scale) — 32

Stress test

weight histogramgrid levels of block 0

identity (lossless)quantize → dequantize

RMS error

—

Signal/noise

—

Effective bpw

—

Overhead

—

Read the staircase

The lower panel is the honest picture. The diagonal grey line is what you wanted; the blue staircase is what you get. Each flat tread is a set of distinct weights made identical. At 4 bits the treads are narrow and the staircase hugs the line. At 2 bits, whole regions collapse into one value. At 1 bit the function has two outputs and the model is destroyed.

Why one outlier is a catastrophe

Tap inject outlier above — one weight at 20σ — and watch the RMS error explode while the grid levels spread uselessly across empty space.

This is the central practical problem. If a single weight in a 4096×4096 tensor is 20× larger than the rest, then s = max|w|/7 is 20× too big and every ordinary weight rounds to −1, 0, or +1. You paid for 4 bits and received about 1.6.

The fix is embarrassingly simple: don't share a scale across the whole tensor. Chop the weights into small contiguous blocks, each with its own scale. The outlier now poisons its own 32 neighbours instead of 16 million.

// llama.cpp Q4_0 — the simplest real block format
typedef struct {
    ggml_half d;          // 1 × FP16 scale     →  16 bits
    uint8_t   qs[32/2];   // 32 × 4-bit weights → 128 bits
} block_q4_0;             // 18 bytes for 32 weights

// bits per weight = (32×4 + 16)/32 = 4.5 bpw
// reconstruction:  w = d × (q − 8),  q ∈ [0,15]

Note the accounting. The payload is 4.0 bpw, but the FP16 scale adds 16 bits over 32 weights — +0.5 bpw. That overhead is the price of outlier robustness, and it's the number every format in this series is negotiating over.

Go back to the widget and drag block size with the outlier injected. At 4096 (per-tensor) the error is catastrophic. At 32 it is barely worse than the clean case. That fix costs exactly 0.5 bits per weight.

The overhead spiral, and three escapes

Smaller blocks are better for accuracy, worse for size. At block size 16 with an FP16 scale you pay +1.0 bpw — a quarter of your budget at 4 bits, half of it at 2 bits. Three ways out:

Quantize the scales themselves. Store block scales as 6-bit integers with one FP16 super-scale per 256 weights. That's what GGUF k-quants do.
Double quantization. QLoRA's version: quantize the FP32 block scales to 8-bit with their own scale per 256 blocks.
Reshape the grid so it fits the data better without needing a tight scale. That's NF4.

// QLoRA double quantization, block=64, NF4
naive : 4 + 32/64          = 4.500 bpw
DQ    : 4 + 8/64 + 32/2048 = 4.127 bpw
                             // ~0.37 bpw saved = ~3 GB on a 70B model

The next chapter takes all three of those and builds real file formats out of them.