Where a GPU's time goes
Primer page 2. The surprising fact behind the whole challenge: when a big model serves one user, the GPU is mostly waiting for memory, not calculating.
A kitchen with a distant warehouse
Picture a kitchen with 132 cooks. They are extremely fast with a knife. The ingredients, though, are in a warehouse down the road, and there is one road. However fast the cooks are, dinner cannot be ready before the trucks arrive.
The cooks are the GPU's processors. The ingredients are the model's weights, the billions of learned numbers. The road is memory bandwidth. Most of this challenge is about the road, not the knives.
Four places a number can be
The weights one layer needs for one step come to 160 to 450 MB. They fit only in the first box. So every layer, on every step, has to pull its weights down that one road again. The 50 MB cache cannot hold them, and by the time the same layer comes round on the next step, 61 other layers have pushed everything out.
132 workers, in teams
- An H200 has 132 streaming multiprocessors (SMs). Think of each as one cook's station.
- A kernel is cut into blocks, and each block runs on one SM. If a kernel has 132 blocks, every station gets one.
- Inside a block, threads work in teams of 32 called warps. All 32 do the same instruction at the same moment.
- Each SM contains tensor cores, circuits that multiply small matrices in one go. They are the fast knives.
All 132 stations share the one road. If only a few stations ask for data, the road is half empty and time is lost. Getting every station to ask at once, all the time, is a recurring theme.
Fetching versus arithmetic
Every piece of work has two possible limits: how long it takes to fetch the numbers, and how long it takes to do the arithmetic on them. Whichever is larger sets the time. Pick one of the model's matrices and drag the number of tokens.
At one token, fetching the pooler takes about 15 millionths of a second and the arithmetic takes a tiny fraction of one. The cooks work for a blink and wait for the rest. Only past roughly 230 tokens does the arithmetic catch up.
The arithmetic estimate uses the tensor cores' rated peak of about 989 trillion operations a second. Real kernels reach less.
The life of one kernel
Even a perfect kernel does not run at full speed from its first instant to its last.
- Launch. Telling the GPU to start costs a few microseconds.
- Ramp. The first bytes take about 4 µs to arrive. Until then nobody can work.
- Stream. The good part: bytes flowing at the road's full speed.
- Tail. Some stations finish before others, and the kernel is not done until the last one is.
- Gap. The next kernel cannot begin until this one is fully finished.
When the whole kernel lasts 10 or 20 µs, those edges are a large share of it. One layer is a chain of a dozen or more kernels, a step is 62 layers, and the SGLang team counted about 1,450 launches in a single step on a newer GPU. A lot of the contest's gains came from trimming edges, not from speeding up the middle.
Two tools against the edges
- CUDA graphs. Instead of the server issuing each launch one by one, it records the whole sequence once and replays it. That removes most of the cost on the server's side.
- Programmatic dependent launch (PDL). Lets the next kernel's stations start getting ready, for instance by ordering their weights, while the previous kernel is still finishing. The gap and part of the ramp disappear. Page 5 has the picture.
Waiting on each other
The other way to lose time is coordination. When stations must agree on something ("is everyone done?"), a message has to cross the chip. Measured on the H200, such a signal took about 1.4 µs when the chip was quiet and over 6 µs while the road was busy. Adding up many small numbers into one shared total has a similar cost. Good kernels are arranged so the stations that are on the critical path never wait for anyone.
Cold and warm
Data already sitting in the cache is "warm" and comes back quickly. The contest's kernel judge deliberately emptied the cache before every timing, so everything started "cold". A real server does not do that, but with hundreds of megabytes flowing through a 50 MB cache per layer, the weights are cold on every step anyway. Small things that get reused, such as tables, counters and the kernel's own code, are a different matter, and are worth keeping warm.
- At small batch the limit is bytes per second, not arithmetic.
- The floor for any job is its bytes divided by the road's speed. That is what "floor" means on the plan pages.
- Short kernels lose a large share of their time to launch, ramp, tail and gaps.
- Fewer bytes, fewer edges, no waiting: the three levers from page 1, now with reasons.
Figures: memory sizes and worker count are the H200's published ones; the measured road speed, ramp and signal costs are from participants' measurements collected on the methodology page; the launch count is from the SGLang team's megakernel post. Next: the model, one layer at a time.