The kernel tricks, one idea at a time
Primer page 5. The plan pages are full of names: stream-K, TMA ring, PDL, split-KV. Each is a simple idea. Here they are as a problem, the idea, and the catch.
The two basic jobs
Almost everything a layer does is one of two jobs. Multiply a big table of weights by one token's list of numbers (the plan calls this GEMV). Or multiply it by many tokens' lists at once (GEMM). The tensor cores from page 2 do this in small rectangular bites called tiles. A kernel's work is: cut the job into tiles, hand tiles to the 132 workers, fetch each tile's weights, multiply, and put the results together.
Every trick below improves one of those verbs: cut, hand out, fetch, multiply, put together.
1. Deal the work out evenly
Problem. Tiles are handed out one per worker, in rounds. If the number of tiles is not a neat multiple of 132, the last round leaves most workers idle while a few finish. Drag the slider just past 132.
Idea. Stop dealing in whole tiles. Add up all the work and give every worker an equal share, even if that means two workers each do half of the same tile.
Nine tiles, four workers: the example from the paper that introduced the idea.
Names on the plan. Cutting each tile's inner loop into a fixed number of pieces is split-K. Giving every worker an exactly equal share of the whole job is stream-K. The uneven last round is called wave quantization.
Catch. When two workers each compute half of one answer, the halves must be added together afterwards. Doing that through shared memory, one number at a time, was measured costing up to 77 µs in one design: more than the work it saved. The good versions add the halves somewhere private.
2. Keep deliveries in flight
Problem. If a worker asks for weights, waits, uses them, then asks for more, the road sits empty while it works. And an order placed now takes about a microsecond to arrive.
Idea. Order ahead. Keep a small ring of slots on the workbench. One team does nothing but place orders, always several slots ahead. The other teams take full slots and do the arithmetic. The road stays full and nobody waits.
Names on the plan. The ordering machinery on this GPU is the TMA. Splitting teams into those who fetch and those who compute is warp specialization. "Bytes in flight" is how much has been ordered and not yet arrived: about 33 KB per worker keeps the road full.
Catch. The workbench is small (about 228 KB), and slots use it up. Past four slots, nobody measured a gain. And a consumer that is too slow leaves slots full, which stalls the orders behind them.
3. Do several jobs in one pass
Problem. Multiply, then normalise, then add to the running state: as three kernels, that is three launches and three trips over the same data.
Idea. Do the small follow-up jobs at the end of the big one, while the numbers are still in hand. The plan calls this fusing, and the tail end of a kernel where it happens the epilogue.
Catch. Fusing is not always a win. If two kernels were already overlapping nicely, gluing them into one can remove the overlap. One team measured losses of 2 to 4% from fusions that looked obviously good.
4. Start the next kernel early
Problem. Page 2's timeline: each kernel ends with a tail and a gap, and the next begins with a ramp.
Idea. Weights never depend on the tokens. So the next kernel can order its weights while the current one is still finishing, and only wait for the current kernel's results at the last moment.
Names on the plan. This is PDL (programmatic dependent launch). Taken further, one kernel that never exits and handles a whole layer is a persistent kernel or megakernel.
Catch. There is one road. Ordering early competes with whatever the current kernel is still fetching, and several teams measured early ordering making things slower. It pays only behind a kernel that is not using the road much. The megakernel won on a newer GPU at one user; on this GPU a chain of small kernels beat it.
5. Do not make workers wait for each other
Problem. "Is everyone finished?" is an expensive question: 1.4 µs on a quiet chip, over 6 µs on a busy one. Adding many workers' results into one shared total is slow for the same reason.
Idea. Arrange the work so the question is rarely asked. Use a simple counter that the last worker to arrive notices, so only one worker ever waits. Let neighbouring workers share a workbench directly (the plan calls a group of such neighbours a cluster) so they can combine results without going through main memory.
Catch. Each of these has a cost of its own, and the measurements are full of versions that lost. The reliable rule is only this: no waiting on the path that sets the finish time.
6. Unpack the 4-bit weights on the way in
Problem. The experts' weights arrive as 4-bit codes and must become 16-bit numbers before the tensor cores can use them. Unpacking is ordinary integer work, and on this GPU it does not overlap with the tensor cores: a worker does one or the other.
Idea. Unpack each weight once and feed it straight to the tensor cores, never writing the unpacked form anywhere. Then count instructions: one team got the unpacking loop from 20.1 instructions per word down to 18.4 by reordering the stored data, and that alone was worth a few percent.
Catch. On the newest GPUs the tensor cores read 4-bit weights directly and none of this is needed. On the H200 it is the heart of the expert kernel.
7. Attention in chunks
Problem. A full layer must read notes on 80,000 earlier tokens. One worker doing that alone would take forever, and the weighting step (called softmax) seems to need all the scores before it can produce any answer.
Idea. Cut the history into chunks and give each worker one. Each produces a partial answer plus two small bookkeeping numbers. A neat piece of algebra lets partial answers be merged exactly, in any order.
Top: the history, cut into chunks. Middle: one worker per chunk. Bottom: the merge.
Names on the plan. Chunking the history is split-KV. The algebra is the "online softmax" from the FlashAttention papers. Answering eight questions per pass over a pile (page 3) is the GQA tile.
Catch. The merge is again a "put together" step, with the costs of section 5. And when users have histories of very different lengths, chunks must be dealt evenly across all of them, which is section 1 again.
8. One kernel per size, and its cost
At 8 tokens the job is all fetching; at 256 it is half arithmetic. The best kernel for one is poor at the other, so the top entries built a separate kernel for each batch size the judge timed. That worked for a judge that timed six sizes. A real server sees every size in between, and a kernel tuned for exactly 32 tokens can stumble at 40. This is one of the open experiments on the plan.
How the wins were actually found
- Measure the floor first. Write a do-nothing kernel that only fetches the same bytes. Its time is the best any real kernel can do. Several teams discovered their kernel was already at the floor and stopped polishing it.
- Draw the timeline. Record when each worker starts, waits and finishes. Nearly every large win came from spotting one slow stretch on that picture.
- Compare in pairs. Run old and new back to back, many times, on the same GPU. Run-to-run noise is about 1.5 to 2%, the same size as most improvements.
- Expect most ideas to fail. The most complete public record lists about 460 experiments. A small fraction survived.
- Deal work in equal shares, not whole tiles.
- Order weights ahead so the road is never empty.
- Fuse small jobs into big ones, but check that it helped.
- Start the next kernel's fetch early, but only when the road has room.
- Keep waiting off the path that sets the finish time.
- Every one of these has a measured case where it lost. That is why the plan insists on measuring.
Measurements quoted here are collected, with sources, on the methodology and prior-art pages. The even-shares example is from the Stream-K paper (Osama, Merrill and Cecka, 2023). For more on fusing, see Fused kernels on tensor cores on this site. Next: serving and the score.