Serving, the frontier and the score

Primer page 6. A fast kernel is only half the story. The prize goes to whoever makes the whole server better, and "better" has a precise meaning that changes what is worth doing.

Two numbers that pull against each other

Think of a bus. With one passenger it leaves at once and drives straight there. With forty it stops more and takes longer, but it moves forty people. A model server makes the same trade every moment.

Page 3 gave the reason they trade. A step for 32 users takes about three times as long as a step for one, because the expensive weight fetch is shared. Each user gets a third of the speed, and the machine produces ten times as much.

Users at onceStep timeSpeed each user seesTotal output
116.6 ms301 tokens/s301 tokens/s
419.9 ms2511,003
1633.3 ms1502,406
3251.0 ms983,136

Illustrative: a straight line through stock's GPU time per step, and an assumed five tokens accepted per step. Real figures include more overhead, but the shape is right.

The frontier, and what the contest scores

Plot those rows with speed along the bottom and output up the side, and you get a curve: the frontier. Every point on it is a choice of how many users to carry. A better server has a curve that sits higher: more output at the same speed.

The contest's score asks exactly that, at four points. In plain words: at the speed stock gives 1, 4, 16 and 32 users, how much more does your server put out? The four answers are ratios, and the score is their geometric mean.

Here the step is split into a fixed part, which costs the same however many users there are (fetching the weights, launches, the draft), and a per-user part, which grows with each user added (attention over each history, per-token arithmetic). Speed up each part and watch the four ratios.

Score
stock serverpatched server
Throughput: tokens per second per GPU Per-user speed: tokens per second for each user

Hover the chart to read one load.

Stock's loadSpeed per userStock output per GPUUsers you carry at that speedYour output per GPURatio

Three things to try

  1. Set both sliders to 1.2×. The step is only 20% faster, yet the score is about 1.9. Look at the first row: you can carry four users at the speed stock gives one. Adding a user is cheap, so a small saving buys several.
  2. Fixed part 1.2×, per-user part 1×. The score stays near 1.6, and the gain is concentrated at the low loads.
  3. Fixed part 1×, per-user part 1.2×. Every ratio is exactly 1.2 and so is the score.
The lesson. If the score works this way, time that does not grow with users is worth far more than time that does. Fewer weight bytes, fewer launches and a faster draft are all fixed cost. This is why the plan's serving page now ranks them where it does.
Two cautions. The straight-line model is a simplification, and it ignores any limit on how many users the test will actually add. And the organizers' exact method is not public: this is the plain reading of their one-sentence rule. The first official score shows whether it holds. If the four ratios come back similar to each other, it does not.

The wait before the first word

Speed while streaming is one thing. The pause before anything appears is another: time to first token, or TTFT. It is mostly the time to read the incoming message (prefill), plus any time spent queueing behind other people's work.

As far as the published rule goes, the contest does not reward a shorter TTFT. It only forbids a long one: no more than twice stock's at each load. That makes TTFT a budget. A server may be a little slower to start if that buys faster streaming, as long as it stays inside the limit.

Not re-reading the conversation

In a long chat or a coding session, every new message arrives with the entire history attached. Reading it all again each time would be absurd, so the server keeps the notes from last time (the KV cache from page 3) and reads only what is new.

must be read nownotes already kept
Turn 1read everything: about 16,000 tokens Turn 2reuse the notes, read about 750 new tokens Turn 3and again

This is called prefix caching. It only works if the start of the new message matches what was kept, token for token, and if the kept notes have not been thrown away to make room. In a workload made of long multi-turn sessions, as the contest's is, the cache does most of the work of keeping TTFT short.

When reading and writing collide

All users share the same eight GPUs. When a new message arrives and has to be read, that work competes with everyone else's streaming replies.

steps writing everyone's repliesreading one new message
everyone's reply pauses here

The server can cut a long read into smaller pieces (chunked prefill) and slot them between writing steps, so nobody's stream stalls for long. Smaller pieces mean smoother streaming for others and a longer wait for the newcomer. How to strike that balance is a server policy, not a kernel, and it matters most with 16 or 32 users.

The rules that keep it honest

The last three mean that a tidy, general improvement can beat a faster but sprawling one.

Knowing whether a change helped

What to take away.

The rules are from the official summary linked on the serving page, which also has the plan built on them. The local test rig is on serving harness & runs. Back to the start of the primer.