Number formats and what "lossless" means

Primer page 4. If time is bytes divided by road speed, the size of each number is the most direct lever there is. This page is about making numbers smaller, and about the line between shrinking data and changing it.

How a computer writes a number

Scientists write big and small numbers as digits times a power of ten: 6.02 × 10²³. Computers do the same in base two. A number is stored as three parts:

Formats differ only in how many bits each part gets. Fewer bits means fewer distinct values, and less to fetch.

signexponent (how big)mantissa (the digits)
FP3232 bits: 1 + 8 + 23 BF1616 bits: 1 + 8 + 7 FP88 bits: 1 + 4 + 3 FP44 bits: 1 + 2 + 1

BF16 is FP32 with most of the digits cut off but the full range of sizes kept. That is why it is popular for model weights.

FormatBitsWhere this model uses it
FP3232The token's running state, the router, and running totals during arithmetic.
BF1616The four fixed matrices in each layer, and the values passed between parts.
FP88The notes attention keeps on earlier tokens (the KV cache).
NVFP4about 4.5The experts' weights: the bulk of the model.

How 4-bit weights can work at all

Four bits give sixteen patterns. In the FP4 layout they spell out just eight sizes, each with a plus or minus: 0, 0.5, 1, 1.5, 2, 3, 4 and 6. On its own that would be hopeless for weights that range from thousandths to tens.

The fix is a shared scale. Weights are grouped into blocks of 16. Each block stores one extra 8-bit number, its scale, chosen so that the block's largest weight lands on 6. Every weight in the block is then stored as the nearest of the eight sizes, and decoded as size × scale. Sixteen 4-bit codes plus one 8-bit scale is 4.5 bits per weight. NVIDIA calls this NVFP4. There is also one more scale for the whole matrix, which the picture leaves out.

Block scale: Typical error:
true weight ÷ scalewhat gets stored
−6−4−3−2−1012346
True weightStored sizeDecoded as

First six of the sixteen shown. Notice the allowed sizes are packed closely near zero, where most weights are, and sparsely at the edges.

This loses precision, and the people who made the model accepted that when they published a 4-bit version. From here on, those 4-bit codes are the model. The contest's job is to run them faster, not to round them again.

Lossless: smaller, but exactly the same

"Lossless" means that after unpacking, every bit is what it was. A zip file is lossless; a blurry photo is not. The contest allows any repacking of the weights as long as it is lossless. Three tricks from the plan pages:

1. Sixteen-bit weights in twelve bits

A BF16 number spends 8 of its 16 bits on the exponent, which can describe sizes from unimaginably small to unimaginably large. A trained model's weights are nearly all about the same size, so in practice nearly all of them use one of about 15 exponent values. Instead of storing the 8-bit exponent, store which of the 15 it is. That takes 4 bits.

BF16, 16 bits Packed, 12 bits only about 15 exponent values are common,so store which one: 4 bits in place of 8

The rare weight with an unusual exponent goes on a short side list with its full value. Unpacking gives back the original 16 bits exactly. The result is 25% fewer bytes to fetch for the fixed matrices, which page 2 showed are limited by fetching at small batch.

2. Six-bit scales

The same idea, applied to the 8-bit block scales of the 4-bit weights. Each matrix turns out to use only about 33 distinct scale values, so a 6-bit index into a small table is enough.

3. Unpacking 4-bit weights exactly

To multiply, the 4-bit code and its scale are combined into an ordinary 16-bit number. The code has at most 2 significant binary digits and the scale at most 4, so their product has at most 6. BF16 holds 8. Nothing is rounded: the 16-bit number is exactly the value the 4-bit format meant.

What is not lossless

Tempting shortcutWhy it changes the answer
Keep running totals in 16 bits, not 32Adding thousands of small numbers into a short total drops digits at every step.
Use a faster, rougher formula for expAttention's weighting depends on it; small errors shift which notes get blended in.
Skip index cards that probably do not matter"Probably" is a guess. The model reads all of them.
Run 7 experts when the router chose 8The eighth contributed something, however small.
Round 16-bit weights to 12 bits (rather than repack them)That is a different model.
The rules moved here. Late in the kernel round the organizers allowed a few of these shortcuts where the error stayed below 16-bit precision. The serving round says something stricter: nothing below what stock SGLang does. The plan pages flag every trick that leaned on the looser rule.

Same arithmetic, different last digit

One subtlety. Even honest, full-precision arithmetic is not perfectly repeatable. Adding the same numbers in a different order can change the last digit, because each addition rounds. A kernel that splits a sum across 132 workers adds in a different order from one that does not.

So the serving round does not demand identical bits. It checks that a patched server's answers agree with stock's about as well as stock agrees with itself from run to run. Reordering sums passes. Lower precision does not.

What to take away.

Format details follow NVIDIA's description of NVFP4. The 12-bit and 6-bit figures and the exactness argument are from the bf16_proj and moe_w4a16 pages. For a longer tour, see How LLM quantization works on this site. Next: the kernel tricks.