Serving, the frontier and the score
Primer page 6. A fast kernel is only half the story. The prize goes to whoever makes the whole server better, and "better" has a precise meaning that changes what is worth doing.
Two numbers that pull against each other
Think of a bus. With one passenger it leaves at once and drives straight there. With forty it stops more and takes longer, but it moves forty people. A model server makes the same trade every moment.
- Per-user speed: how many tokens per second each person sees. The contest calls this interactivity.
- Throughput: how many tokens per second the whole machine produces, across everyone. This is what pays for the hardware.
Page 3 gave the reason they trade. A step for 32 users takes about three times as long as a step for one, because the expensive weight fetch is shared. Each user gets a third of the speed, and the machine produces ten times as much.
| Users at once | Step time | Speed each user sees | Total output |
|---|---|---|---|
| 1 | 16.6 ms | 301 tokens/s | 301 tokens/s |
| 4 | 19.9 ms | 251 | 1,003 |
| 16 | 33.3 ms | 150 | 2,406 |
| 32 | 51.0 ms | 98 | 3,136 |
Illustrative: a straight line through stock's GPU time per step, and an assumed five tokens accepted per step. Real figures include more overhead, but the shape is right.
The frontier, and what the contest scores
Plot those rows with speed along the bottom and output up the side, and you get a curve: the frontier. Every point on it is a choice of how many users to carry. A better server has a curve that sits higher: more output at the same speed.
The contest's score asks exactly that, at four points. In plain words: at the speed stock gives 1, 4, 16 and 32 users, how much more does your server put out? The four answers are ratios, and the score is their geometric mean.
Here the step is split into a fixed part, which costs the same however many users there are (fetching the weights, launches, the draft), and a per-user part, which grows with each user added (attention over each history, per-token arithmetic). Speed up each part and watch the four ratios.
Hover the chart to read one load.
| Stock's load | Speed per user | Stock output per GPU | Users you carry at that speed | Your output per GPU | Ratio |
|---|
Three things to try
- Set both sliders to 1.2×. The step is only 20% faster, yet the score is about 1.9. Look at the first row: you can carry four users at the speed stock gives one. Adding a user is cheap, so a small saving buys several.
- Fixed part 1.2×, per-user part 1×. The score stays near 1.6, and the gain is concentrated at the low loads.
- Fixed part 1×, per-user part 1.2×. Every ratio is exactly 1.2 and so is the score.
The wait before the first word
Speed while streaming is one thing. The pause before anything appears is another: time to first token, or TTFT. It is mostly the time to read the incoming message (prefill), plus any time spent queueing behind other people's work.
As far as the published rule goes, the contest does not reward a shorter TTFT. It only forbids a long one: no more than twice stock's at each load. That makes TTFT a budget. A server may be a little slower to start if that buys faster streaming, as long as it stays inside the limit.
Not re-reading the conversation
In a long chat or a coding session, every new message arrives with the entire history attached. Reading it all again each time would be absurd, so the server keeps the notes from last time (the KV cache from page 3) and reads only what is new.
This is called prefix caching. It only works if the start of the new message matches what was kept, token for token, and if the kept notes have not been thrown away to make room. In a workload made of long multi-turn sessions, as the contest's is, the cache does most of the work of keeping TTFT short.
When reading and writing collide
All users share the same eight GPUs. When a new message arrives and has to be read, that work competes with everyone else's streaming replies.
The server can cut a long read into smaller pieces (chunked prefill) and slot them between writing steps, so nobody's stream stalls for long. Smaller pieces mean smoother streaming for others and a longer wait for the newcomer. How to strike that balance is a server policy, not a kernel, and it matters most with 16 or 32 users.
The rules that keep it honest
- Lossless. No lower precision, no approximations, no skipped work beyond what stock does (page 4).
- Agreement. The patched server's answers must match stock's about as closely as stock matches itself.
- Memory. At most 2 GB more per GPU than stock. A second copy of the weights in a faster layout will not fit.
- Size. The whole change must be under 400 KB of source code.
- Review. SGLang's maintainers judge whether the change is something they would accept into the project. Code that special-cases the test, for example by checking for exactly 16 users, can be disqualified.
- Held-out reruns. The top entries are run again on conversations nobody has seen.
The last three mean that a tidy, general improvement can beat a faster but sprawling one.
Knowing whether a change helped
- Compare like with like. Same machine, same conversations, old and new run alternately.
- Know the noise. Run stock against itself. If two identical runs differ by 2%, a 1% gain is not a finding.
- Look at each load, not only the average. A geometric mean multiplies the ratios and takes the root, so it cannot hide a weak one: three good loads and one at 1.0 score well below four good ones.
- Test sizes the contest does not. If 8 or 24 users is worse than before, the change was fitted to the test.
- Decide locally. There is one official scored run a day. It confirms; it cannot explore.
- More users means each is slower and the machine does more. The frontier is that trade drawn as a curve.
- The score asks how much more you produce at stock's speeds.
- Under the plain reading, fixed cost per step is worth much more than per-user cost.
- TTFT is a limit, not a goal. Prefix caching and chunked prefill manage it.
- Rules on memory, size and review reward small, general changes.
The rules are from the official summary linked on the serving page, which also has the plan built on them. The local test rig is on serving harness & runs. Back to the start of the primer.