Importance matrices and codebooks

Two ideas that make everything below 4 bits possible: measure which weights matter, and stop quantizing them one at a time.

Minimise output error, not weight error

Every scheme so far has silently optimised the same thing: Σ (w − ŵ)². Make the weights close to the originals. That treats all weights as equally consequential — and they are emphatically not.

What matters is what the layer outputs. For a linear layer y = Wx:

Δy = (W − Ŵ) x

// for one output neuron:
Δy_k = Σ_i (w_ki − ŵ_ki) · x_i

// take the expectation over real input data:
E[Δy_k²] ≈ Σ_i (w_ki − ŵ_ki)² · E[x_i²]
                                  ↑
                        the importance matrix

There it is. A weight's contribution to output error is scaled by the mean squared magnitude of the activation it multiplies. A weight in a column that always sees near-zero activations can be mangled freely. One that sees huge activations must be preserved.

llama.cpp's llama-imatrix does exactly this: run a calibration corpus through the full-precision model, accumulate E[xᵢ²] for every input channel of every linear layer, write it to a file. The quantizer then minimises the weighted objective:

minimise  Σ imp_i · (w_i − ŵ_i)²   instead of   Σ (w_i − ŵ_i)²

which changes both the chosen block scale and, for codebook formats, which codeword each group gets.

The calibration data is part of the model now

This is the uncomfortable consequence. The imatrix depends on what you fed the model. Calibrate on Wikipedia and you've optimised for Wikipedia-shaped activations. Then benchmark on Wikipedia perplexity and — surprise — you look excellent. Unsloth flags this directly: quantizers calibrated on Wikipedia-derived data naturally score better on KL-divergence benchmarks that also use Wikipedia.

There's a sharper trap: instruct models have chat templates. A base-model text corpus never emits <|im_start|> or the special role tokens, so the activation statistics around those tokens — the exact tokens gating every conversational turn — come from data that doesn't contain them. Unsloth's calibration set is deliberately built from chat- and code-shaped data, 1.5M+ tokens, hand-curated, applied per model.

Below three bits, scalar rounding stops working

Do the arithmetic on a 2-bit scalar grid. Four levels. Per 32-weight block you still need a scale, so you're at 2.5 bpw before doing anything smart, and reconstructing a Gaussian onto 4 levels gives a signal-to-noise ratio around 14 dB. Quality falls off a cliff.

The escape is vector quantization. Stop quantizing weights one at a time. Take groups of 8 weights, treat each group as a point in 8-dimensional space, and store the index of the nearest entry in a shared codebook.

This wins for two reasons, both easier to see in 2D than 8D:

• Space filling. A scalar grid tiles space with hypercubes. Hypercubes are a bad shape — their corners stick out into low-density territory. Optimal packings (hexagons in 2D, the E₈ lattice in 8D) cover the same region with lower average distortion for the same number of points.
• Density matching. A codebook puts its entries where the data actually is. A product of scalar grids can't — it's forced into a rectangular arrangement even if the joint distribution is a blob.

Same bit budget. Same number of codes. Strictly lower error, just from letting the codes move. In 8 dimensions the gap is much larger — this is Shannon's sphere-packing gain, and it grows with dimension.

How llama.cpp's I-quants use this

The IQ family are codebook formats. Roughly:

• Weights are grouped into small vectors (8 at a time).
• A fixed lookup table of grid points holds the magnitude pattern of the group.
• Separate bits store the signs — sign patterns are near-uniform, so it's cheaper to store them explicitly than to blow up the codebook by 2⁸.
• A superblock scale and the importance matrix govern reconstruction.

This is why the format table marks IQ quants as requiring an imatrix. There's no sensible way to pick "nearest codeword" without knowing which coordinates matter — unweighted nearest-neighbour in 8D at 2 bits produces garbage. The codebook and the calibration statistics are a package deal.

1-bit, 1.58-bit, and what those names mean