Shaping the grid: NF4 and GGUF k-quants

Equal-width grid steps are the wrong choice for a bell curve. Here's what the two dominant real-world formats do instead — down to the individual byte.

NF4: put the levels where the weights are

Uniform quantization spaces its 16 levels evenly from −max to +max. But weights are Gaussian: almost all of them bunch near zero, and the levels out at ±max serve a handful of values. You've allocated scarce codes to a region where nothing lives.

NormalFloat4 (NF4), introduced with QLoRA, fixes this with quantile quantization. Instead of equal-width bins it uses equal-probability bins: each of the 16 codes is expected to claim the same number of weights. The level values are the quantiles of a standard normal, rescaled so the extremes land on ±1.

The result is information-theoretically optimal if your weights are truly normal — every code carries the same information, so no bits are wasted on empty regions.

Look at the density curve behind the levels. The blue NF4 ticks crowd toward the centre where the probability mass is and stretch out into the tails where weights are rare. The orange uniform ticks are indifferent to the data.

The actual NF4 codebook — 16 constants baked into bitsandbytes:

NF4 = [−1.0000, −0.6962, −0.5251, −0.3949, −0.2844, −0.1848,
       −0.0911,  0.0000,  0.0796,  0.1609,  0.2461,  0.3379,
        0.4407,  0.5626,  0.7230,  1.0000]

// asymmetric on purpose: 8 negative + zero + 7 positive.
// exact zero must be representable — padding and masked
// weights have to dequantize to exactly 0.0

This is the format behind load_in_4bit=True and the "4-bit bitsandbytes" checkpoints on Hugging Face. It's what QLoRA fine-tuning runs on top of.

Inside a Q4_K superblock

GGUF is the container used by llama.cpp, Ollama, LM Studio and friends. Its "k-quants" are where block-wise scaling gets genuinely clever: scales are themselves quantized, in a two-level hierarchy.

A Q4_K superblock covers 256 weights, split into 8 sub-blocks of 32. Each sub-block gets its own 6-bit scale and 6-bit minimum. Those 6-bit values are then scaled by two FP16 super-scales for the whole superblock. Tap the byte map:

// reconstruction, weight i in sub-block j
scale_j = d    × scales6[j]      // FP16 × 6-bit int
min_j   = dmin × mins6[j]
w[i]    = scale_j × q[i] − min_j // q ∈ [0,15]

// 256×4 + 8×6 + 8×6 + 16 + 16
// = 1024 + 96 + 32 = 1152 bits / 256 = 4.5 bpw

Compare that to naive per-32 FP16 scales, which would cost 128 bits of metadata for symmetric blocks alone. Here you get the accuracy of fine-grained asymmetric blocks at roughly the same metadata cost. That's the k-quant contribution.

What the suffixes mean

A filename like model-Q4_K_M.gguf encodes three things: nominal bit width (4), family (K = k-quant), and mixing policy (M = medium). The policy is the important part.

Q4_K_M does not put every tensor in Q4_K. It promotes specific tensors — historically the attention value projections (attn_v) and the feed-forward down-projections (ffn_down) — to Q6_K, because empirically those hurt most when crushed. Q4_K_S skips the promotion.

The format table

These are structural costs. A real file's average bpw is a weighted mix, because the quantizer promotes some tensors and leaves token embeddings and the output head at higher precision.