Where a GPU's time goes

Primer page 2. The surprising fact behind the whole challenge: when a big model serves one user, the GPU is mostly waiting for memory, not calculating.

A kitchen with a distant warehouse

Picture a kitchen with 132 cooks. They are extremely fast with a knife. The ingredients, though, are in a warehouse down the road, and there is one road. However fast the cooks are, dinner cannot be ready before the trucks arrive.

The cooks are the GPU's processors. The ingredients are the model's weights, the billions of learned numbers. The road is memory bandwidth. Most of this challenge is about the road, not the knives.

Four places a number can be

Main memory (HBM)141 GBone road: ~4.3 TB/s L2 cacheabout 50 MBshared by all Shared memoryabout 228 KB per workera private workbench Registersa few KBin hand large, far, slow to reachtiny, near, instant

The weights one layer needs for one step come to 160 to 450 MB. They fit only in the first box. So every layer, on every step, has to pull its weights down that one road again. The 50 MB cache cannot hold them, and by the time the same layer comes round on the next step, 61 other layers have pushed everything out.

132 workers, in teams

All 132 stations share the one road. If only a few stations ask for data, the road is half empty and time is lost. Getting every station to ask at once, all the time, is a recurring theme.

Fetching versus arithmetic

Every piece of work has two possible limits: how long it takes to fetch the numbers, and how long it takes to do the arithmetic on them. Whichever is larger sets the time. Pick one of the model's matrices and drag the number of tokens.

fetch the weightsdo the arithmetic
Fetch the weightsDo the arithmetic 15.6 µs 0.07 µs

At one token, fetching the pooler takes about 15 millionths of a second and the arithmetic takes a tiny fraction of one. The cooks work for a blink and wait for the rest. Only past roughly 230 tokens does the arithmetic catch up.

This is why one user is the hard case. The weights have to be fetched once per layer whether there is 1 token or 200. With one user that whole cost lands on one token. With many users it is shared, which is why batching makes each token cheaper, and why kernels for small batches look nothing like kernels for large ones.

The arithmetic estimate uses the tensor cores' rated peak of about 989 trillion operations a second. Real kernels reach less.

The life of one kernel

Even a perfect kernel does not run at full speed from its first instant to its last.

launchrampsteady stream at full road speedtailgaplaunchrampnext kernel time

When the whole kernel lasts 10 or 20 µs, those edges are a large share of it. One layer is a chain of a dozen or more kernels, a step is 62 layers, and the SGLang team counted about 1,450 launches in a single step on a newer GPU. A lot of the contest's gains came from trimming edges, not from speeding up the middle.

Two tools against the edges

Waiting on each other

The other way to lose time is coordination. When stations must agree on something ("is everyone done?"), a message has to cross the chip. Measured on the H200, such a signal took about 1.4 µs when the chip was quiet and over 6 µs while the road was busy. Adding up many small numbers into one shared total has a similar cost. Good kernels are arranged so the stations that are on the critical path never wait for anyone.

Cold and warm

Data already sitting in the cache is "warm" and comes back quickly. The contest's kernel judge deliberately emptied the cache before every timing, so everything started "cold". A real server does not do that, but with hundreds of megabytes flowing through a 50 MB cache per layer, the weights are cold on every step anyway. Small things that get reused, such as tables, counters and the kernel's own code, are a different matter, and are worth keeping warm.

What to take away.

Figures: memory sizes and worker count are the H200's published ones; the measured road speed, ramp and signal costs are from participants' measurements collected on the methodology page; the launch count is from the SGLang team's megakernel post. Next: the model, one layer at a time.