Number formats and what "lossless" means
Primer page 4. If time is bytes divided by road speed, the size of each number is the most direct lever there is. This page is about making numbers smaller, and about the line between shrinking data and changing it.
How a computer writes a number
Scientists write big and small numbers as digits times a power of ten: 6.02 × 10²³. Computers do the same in base two. A number is stored as three parts:
- a sign: plus or minus;
- an exponent: roughly how big the number is;
- a mantissa: the digits themselves.
Formats differ only in how many bits each part gets. Fewer bits means fewer distinct values, and less to fetch.
BF16 is FP32 with most of the digits cut off but the full range of sizes kept. That is why it is popular for model weights.
| Format | Bits | Where this model uses it |
|---|---|---|
| FP32 | 32 | The token's running state, the router, and running totals during arithmetic. |
| BF16 | 16 | The four fixed matrices in each layer, and the values passed between parts. |
| FP8 | 8 | The notes attention keeps on earlier tokens (the KV cache). |
| NVFP4 | about 4.5 | The experts' weights: the bulk of the model. |
How 4-bit weights can work at all
Four bits give sixteen patterns. In the FP4 layout they spell out just eight sizes, each with a plus or minus: 0, 0.5, 1, 1.5, 2, 3, 4 and 6. On its own that would be hopeless for weights that range from thousandths to tens.
The fix is a shared scale. Weights are grouped into blocks of 16. Each block stores one extra 8-bit number, its scale, chosen so that the block's largest weight lands on 6. Every weight in the block is then stored as the nearest of the eight sizes, and decoded as size × scale. Sixteen 4-bit codes plus one 8-bit scale is 4.5 bits per weight. NVIDIA calls this NVFP4. There is also one more scale for the whole matrix, which the picture leaves out.
| True weight | Stored size | Decoded as |
|---|
First six of the sixteen shown. Notice the allowed sizes are packed closely near zero, where most weights are, and sparsely at the edges.
This loses precision, and the people who made the model accepted that when they published a 4-bit version. From here on, those 4-bit codes are the model. The contest's job is to run them faster, not to round them again.
Lossless: smaller, but exactly the same
"Lossless" means that after unpacking, every bit is what it was. A zip file is lossless; a blurry photo is not. The contest allows any repacking of the weights as long as it is lossless. Three tricks from the plan pages:
1. Sixteen-bit weights in twelve bits
A BF16 number spends 8 of its 16 bits on the exponent, which can describe sizes from unimaginably small to unimaginably large. A trained model's weights are nearly all about the same size, so in practice nearly all of them use one of about 15 exponent values. Instead of storing the 8-bit exponent, store which of the 15 it is. That takes 4 bits.
The rare weight with an unusual exponent goes on a short side list with its full value. Unpacking gives back the original 16 bits exactly. The result is 25% fewer bytes to fetch for the fixed matrices, which page 2 showed are limited by fetching at small batch.
2. Six-bit scales
The same idea, applied to the 8-bit block scales of the 4-bit weights. Each matrix turns out to use only about 33 distinct scale values, so a 6-bit index into a small table is enough.
3. Unpacking 4-bit weights exactly
To multiply, the 4-bit code and its scale are combined into an ordinary 16-bit number. The code has at most 2 significant binary digits and the scale at most 4, so their product has at most 6. BF16 holds 8. Nothing is rounded: the 16-bit number is exactly the value the 4-bit format meant.
What is not lossless
| Tempting shortcut | Why it changes the answer |
|---|---|
| Keep running totals in 16 bits, not 32 | Adding thousands of small numbers into a short total drops digits at every step. |
Use a faster, rougher formula for exp | Attention's weighting depends on it; small errors shift which notes get blended in. |
| Skip index cards that probably do not matter | "Probably" is a guess. The model reads all of them. |
| Run 7 experts when the router chose 8 | The eighth contributed something, however small. |
| Round 16-bit weights to 12 bits (rather than repack them) | That is a different model. |
Same arithmetic, different last digit
One subtlety. Even honest, full-precision arithmetic is not perfectly repeatable. Adding the same numbers in a different order can change the last digit, because each addition rounds. A kernel that splits a sum across 132 workers adds in a different order from one that does not.
So the serving round does not demand identical bits. It checks that a patched server's answers agree with stock's about as well as stock agrees with itself from run to run. Reordering sums passes. Lower precision does not.
- A number is a sign, a size and some digits. Formats trade digits for bytes.
- 4-bit weights work because 16 of them share a scale.
- Lossless repacking exploits the fact that real weights use only a few of the patterns a format allows.
- Fewer bytes fetched is time saved, with no change in results.
- The line the rules draw: repack freely, never round.
Format details follow NVIDIA's description of NVFP4. The 12-bit and 6-bit figures and the exactness argument are from the bf16_proj and moe_w4a16 pages. For a longer tour, see How LLM quantization works on this site. Next: the kernel tricks.