Serving track: where the prizes are
Opened Nov 12, 9 am PT; closes Nov 19, 11:59 pm PT. Private leaderboard. A patch to SGLang v0.5.23 serving Muse Spark Open on one 8×H200 node. Revised Nov 14 against the official rules summary.
python/sglang source, stay within 2 GB per GPU of stock build memory, and pass a maintainer review that can disqualify load-specific code. About seven scored runs remain. So the plan is now tiered: free and small general patches first, to get a safe score on the board, then kernel ports one at a time, each with a byte allowance. Per-concurrency DFlash tuning, the fused all-reduce, expert re-sharding and the megakernel have moved down or out.What is scored
From the official summary (llms.txt, read Nov 14) unless tagged notes, which means a participant's repo: confirm against RULES.md from okc serve init. The full table is on the overview.
- Score: at each load, your tokens/s per GPU at stock's per-user speed, divided by stock's. The geomean over loads is the score, and stock is 1.00.
- Loads: 1, 4, 16 and 32 users. Scored runs cover all four; a check run covers one (16 by default). notes Hidden multi-turn agentic-coding conversations, exactly 220 output tokens per request.
- Gates: lossless (no lower precision, approximation or skipped work beyond stock); TTFT at most 2× stock at each load; build memory at most stock + 2 GB per GPU. notes Outputs must agree with stock about as well as stock agrees with itself; all 8 routed experts, the norms, the attention window and DFlash stay.
- Patch: base commit
297e54fd(v0.5.23), imagelmsysorg/sglang:v0.5.23. Onlypython/sglangsource and JSON configs; nothing undersrt/model_loader/; no renames or binaries; 400 KB at most; no added line over 400 characters. Launch flags change through theflagsfile, except those marked fixed. - Review: maintainers judge mergeability, and that counts toward the winners. Special-casing the workload or the load levels, breaking other models or GPUs, or sprawl can disqualify. Top entries are rerun several times on held-out data.
- Budget: 1 scored run and 3 checks per day, reset at 00:00 UTC; up to 90 minutes of node time per run. Kernel top 50 go first in the queue.
How the kernel work maps onto serving
| Maps | Does not map |
|---|---|
|
|
Where the stock step goes, from the kernel judge's own numbers
SGLang's per-layer times on the judge (georgxzh's run, tabulated on the decode-layer and MoE pages) give a first estimate of stock layer-kernel time per verify step: 46 sliding layers plus 16 full layers.
| Users (verify tokens) | Sliding / full layer (µs) | Layer kernels per step | Extra from the 16 full-attention layers | Stock experts alone, per layer |
|---|---|---|---|---|
| 1 (8) | 266.9 / 270.4 | 16.6 ms | 0.06 ms (0.3%) | 141.9 µs |
| 16 (128) | 523.5 / 694.4 | 35.2 ms | 2.7 ms (8%) | 459.2 µs |
| 32 (256) | 732.0 / 1,086.8 | 51.1 ms | 5.7 ms (11%) | 617.0 µs |
- Experts are about half the stock layer at one user (141.9 of 266.9 µs) and more at 16 and 32. The standalone figures at 128 and 256 tokens can't be subtracted cleanly, because that problem makes this GPU the busiest, but they point well past half.
- Amdahl sketch at one user: if the expert kernel alone ran 2.4× faster (the best public board figure), the sliding layer would go from 266.9 to about 184 µs, a 1.45× gain on layer-kernel time. Larger loads should gain more. This is a bound to test, not a forecast.
- Full attention matters only at 16 and 32 users, and only when contexts are long. The judge used about 80K tokens; real conversations start shorter.
- Caveats: cold L2, communication as identity, busiest-GPU routing. Communication, the draft model and host time come on top. Use the table to sanity-check your own profile, not to replace it.
Step-time model
request(c) ≈ TTFT(c) + (220 / accepted_tokens_per_step) × step(c)
TTFT(c) ≈ queue wait + prefill(new tokens, given the cached prefix)
step(c) ≈ draft(c) + Σ_62 layers [ layer_kernels(c, q_len=8) + comm(c) ] + host/scheduler
per-user speed(c) ≈ accepted_tokens_per_step / step(c) # confirm how the judge defines it
tokens/s per GPU ≈ c × 220 / request(c) / 8
# the score holds per-user speed at stock's value, so a faster step is cashed in as more users served
Measure the breakdown before porting anything: one trace of a step at 1 user and one at 32, CUDA graphs on, split into (a) layer kernels by family, (b) communication, (c) draft, (d) host and gaps, (e) prefill's share of GPU time. Two unknowns decide the plan. The first is communication on stock H200: LMSYS's GB300 megakernel layout has about five sync stops per layer, and stock collectives may cost far more than its 2 µs flags. The second is launch count: LMSYS counted about 1,450 kernel launches per stock step on GB300, and 12.9 ms falling to 5.9 ms with a megakernel at one request.
Patch plan, in tiers
The order is by expected gain per unit of risk and bytes, not by kernel-board speedup. Each row needs a local A/B outside noise at every load it touches before it joins the patch.
Tier 0: no code
| # | Change | Why | EV | Risk |
|---|---|---|---|---|
| 0.1 | Sweep the launch flags that are not marked fixed: scheduler policy, chunked-prefill size, running-request cap, cache settings. | Free, zero patch bytes, and it shows how much of the frontier is configuration. If per-user speed is measured on decode only, TTFT below 2× stock earns nothing, so headroom there can be traded for decode throughput. | Unknown until measured | Low. Keep TTFT at or below about 1.6× stock for noise. |
Tier 1: small general patches (the first scored entry)
| # | Patch | Why / evidence | EV | Rule and merge risk |
|---|---|---|---|---|
| 1.1 | Marlin tile and thread selection on Hopper, by token-count range. | chenwainuo's active serving patches cap the expert tile at 16 through 256 tokens and 32 through 512, and leave larger prefills on stock selection. No numbers published, but it is a few lines on the largest cost in the layer. | Medium | Low. A range rule is general; a rule per benchmark load is not. |
| 1.2 | Fused finalization at small batch for the expert output. | Also in chenwainuo's active set; their stage list ends with a finalizer after the down GEMM. Fewer launches at 1 and 4 users. | Medium | Low. Keep stock's BF16 rounding points. |
| 1.3 | PDL chaining inside the CUDA graph: programmatic stream serialization, with producers triggering after their last weight issue. | +0.9% per layer measured in isolation on the kernel judge. SGLang's own low-latency work launches dependent kernels early the same way. | Medium | Low. |
| 1.4 | Host path: no Python on the step path, overlap scheduler on, graphs captured for the batch sizes that occur. | 220-token requests at one user are latency-sensitive end to end. | Medium | Each captured graph holds memory: watch the +2 GB per GPU cap. |
| 1.5 | Prefill and decode interleave: a general policy for how much prefill may run between decode steps, plus prefix-cache reuse across turns. | Multi-turn conversations share long prefixes. At 16 and 32 users a new turn's prefill stalls everyone's decode. | High at 16 and 32 | Must be a general policy with a sane default, not a setting per load. |
Tier 2: kernel ports, one at a time
| # | Port | Why / evidence | EV | Rule and merge risk |
|---|---|---|---|---|
| 2.1 | Experts: the persistent NVFP4 kernel in place of Marlin at decode and verify token counts, under SGLang's fused-MoE interface (MoE page). | The largest term in the stock layer (table above). 2.42× best public, 2.84× at the board's close. Stock has no MegaMoE on Hopper. | High | Re-layout in place after load: no second copy, nothing in model_loader/. Two or three builds by token range, not one per board shape. FP32 partial sums as in stock. |
| 2.2 | Dense projections: lossless 12-bit GEMVs at 32 tokens and below; stock path above (bf16 page). | Dense GEMVs were 51.5 of 154 µs in an optimized layer against a 34.9 µs floor (alygg77): about 16 µs a layer, 1 ms a step. | High at 1 and 4 | The memory cap allows about 32 MB of duplicated weights per layer, so both encodings of every matrix will not fit. Converting in place slows prefill GEMMs: check TTFT. |
| 2.3 | Attention: full-attention decode and verify only, as a backend for this model (attention page). | Worth up to about 5.7 ms of a 51 ms step at 32 users with long contexts; near nothing at one user. | Medium, conditional on the profile | The highest rule risk: the FP16 and approximate-math paths are kernel-track allowances. Port only the scheduling and gather structure unless RULES.md allows more. |
Tier 3: check first, or leave out
| Item | Earlier rank | Now |
|---|---|---|
| Fused all-reduce + residual + RMSNorm | #4, High | Check whether the v0.5.23 image already ships an all-reduce fusion that works on Hopper for this model. If it is a switch in Python, it becomes Tier 1. If it needs new cross-process GPU code, leave it: the bytes, hang risk and review cost are too high for five days. |
| DFlash tuning per concurrency | #6, High | The speculative flags may be fixed, a policy keyed to the four loads is special-casing, and block verification changes the sampling path that the agreement gate checks. At most one general setting, and only if the flags file leaves it open. |
| Draft model as fused launches | #9, Medium | Only if the profile puts the draft above about a tenth of the step. On GB300 it was 1.16 of 12.9 ms. |
| Expert re-sharding (N-sharded TP) | #8, Medium | Out. It changes the model's parallel layout and the loader's job, and it is too large to review. |
| Per-layer megakernel | #11, Low | Out. LMSYS's version is batch 1 and Blackwell only, and far too large for 400 KB. |
How the score rewards a faster step
Read literally, the rule compares throughput at the same per-user speed. A faster system reaches stock's speed with more users on board, so the ratio for a load is roughly "how many users can I carry at the speed stock gave that many". That makes the four ratios very unequal. A sketch, with stock step time taken as 15.5 ms plus 1.1 ms per user (a line through the judge-derived 16.6, 35.2 and 51.1 ms above):
| Change to the step | Load 1 | Load 4 | Load 16 | Load 32 | Geomean |
|---|---|---|---|---|---|
| Everything 1.2× faster | 3.99 | 1.90 | 1.37 | 1.29 | 1.91 |
| Only the fixed part 1.2× faster | 3.33 | 1.58 | 1.15 | 1.07 | 1.59 |
| Only the per-user part 1.2× faster | 1.20 | 1.20 | 1.20 | 1.20 | 1.20 |
| Everything 2× faster | 15.96 | 5.49 | 2.87 | 2.44 | 4.98 |
- The fixed part of the step is where the leverage is. Cost that does not grow with users (weights read once per layer, launches, communication, the draft, host time) is amplified at loads 1 and 4. Cost that grows per user (attention over each user's history, per-token arithmetic) returns only its own factor.
- The ratios are linked. Stock's speed at one user is matched by your system at 4, 8 or 16 users, so the load-1 ratio is decided by your step time at those larger batches. Speeding up the 32 to 128-token shapes lifts the low-load ratios.
- The TTFT gate may bite here. If TTFT is checked where your system carries those extra users, prefill has to keep up on a busier node, and prefill speed caps the low-load ratios. If it is checked at the same user count, it does not. Find out which.
- This is a model, not the judge. It assumes the score is read along your own speed-versus-throughput curve, a straight-line step time and no limit above 32 users. Your first scored run settles it: a load-1 ratio far above your step speedup means this reading is in force; four similar ratios mean it is not. There is an interactive version on the primer.
Kernel experiments the public ledgers do not cover
Checked against keell0renz's lessons file and alygg77's negative-results and measured-facts files. Every ledger was built on the single-GPU judge, so the gaps are where serving differs from it. All are hypotheses.
| # | Experiment | Why it is open | First test | Odds |
|---|---|---|---|---|
| 1 | Token counts between the board sizes | The public expert kernels were built per scored size (1, 8, 16, 32+) and the judge timed discrete sizes. In serving the running batch moves, because requests in prefill are not decoding. A dedicated tile for 17–32 tokens ran 2.2–3.0 µs slower at d32, so that range is sensitive. | Sweep every multiple of 8 from 8 to 256 on the expert kernel and the dense dispatch. Plot µs against tokens and remove the cliffs. | High that it matters |
| 2 | KV page size and contiguous reads | The boards fixed page size 1 with arbitrary rows, so every key was a 128-byte gather. The only layout experiment in the ledgers was sorting pages inside the kernel, which lost 162–276 µs per full load. Page size is a launch flag: SGLang's guide pairs 64 with long contexts and 1 with maximum multi-turn reuse. | If the flag is open: a bare read skeleton, rows against 16- or 64-token pages, at 32 requests and 80K context. Then a bulk-read path. Cost: coarser prefix matching, roughly up to 63 tokens re-prefilled per turn. | Medium. Full layers at 16 and 32 users only. |
| 3 | Draft model and sampling path | No board covered them. On GB300 the draft was 1.16 ms of a 12.9 ms step, and 8 launches in place of 175 kernels took it to 0.44 ms. | Profile the draft's share at 1 and 32 users. Check whether a general dense GEMV port already covers the draft's projections. | Medium at 1 and 4 users |
| 4 | Evict-first hints on weight streams | LMSYS's megakernel tags weight streams so they do not push small hot data out of L2. The ledgers only say "twin L2 policies" were at optimum or worse, with no numbers, and under the judge's flush. | Tag the bulk weight copies and measure step time inside serving, not on a flushed harness. | Low, but a few lines |
| 5 | The unexplained memory-layout effect | Adding 3 MB of scratch cost 0.5–1.3 µs at d1 and v1 with identical machine code, and the mechanism was never measured. Each GPU streams 10 GB or more of weights per step, so a per-allocation cost would recur in all 62 layers. | One arena with fixed alignment for all re-encoded weights against default allocation, over at least four layouts. | Unknown |
| 6 | Toolchain and compile time | The best public kernels compile native CUDA at start-up, and the serving image is not the kernel image. | In the v0.5.23 image: does that compile path exist, how long do eight ranks take, and can the port ride the JIT route stock already uses? | Blocking. Do it first. |
Tempting, but skip. Prefetching weights into L2 during communication waits or inside one persistent kernel: with phase A fully L2-resident the expert kernel gained only 0.4–0.65 µs of 22. Prefetching experts predicted from the previous step's routes: LMSYS gained about 0.2 ms of a 5.3 ms verify, then found the prefetch was evicting code and ran faster with it off, on a GPU with a larger L2. Keeping old K/V rows warm: 0.2–0.3 µs at 8 tokens or fewer.
Byte budget
Planning allowances, not measurements. Strip dead variants, comments and per-shape builds first, then measure each part's real diff size.
| Part | Allowance | Note |
|---|---|---|
| Experts (2.1) | 160 KB | The part most worth its bytes. |
| Dense projections (2.2) | 80 KB | One GEMV kernel plus the encoder. |
| Attention (2.3) | 90 KB | Spend only if the profile and the rules both allow it. |
| Tier 1, glue, tests | 40 KB | Tests count toward mergeability. |
| Reserve | 30 KB | Stay under about 390,000 bytes in case KB means 1,000. |
Gates before every submission
The full checklist and commands are on Serving harness & runs. In short:
- Agreement: patched against stock falls inside stock's own run-to-run band, greedy and sampled.
- All 8 experts computed, the full attention window, precision unchanged, DFlash on, exactly 220 output tokens.
- TTFT at or below about 1.6× stock at every load; build memory within stock + 2 GB per GPU.
- Patch scope and size: paths, no renames or binaries, under 400 KB, no added line over 400 characters.
- No regression at any load, including one off-ladder load (8 or 24 users) to catch tuning to the four scored loads.
Schedule, Nov 14 → 19, with the run budget
| Day (PT) | Local work | Official runs |
|---|---|---|
| Nov 14 | Pin the rules (okc serve init, RULES.md, fixed flags). Bring up stock v0.5.23 on 8×H200. Stock baselines at 1/4/16/32, twice. Profile a step at 1 and 32 users. | One check at 16 users with a flags-only or no-op patch, to learn what a run reports. |
| Nov 15 | Tier 0 sweep. Tier 1 patches with the agreement gate. | Checks at 1 and 32 users. First scored run once all four loads beat stock locally. |
| Nov 16 | Experts (2.1) behind a general range dispatch. Memory audit. | Checks at 16 and 1. Scored run only if it beats the current entry locally at every load. |
| Nov 17 | Dense projections (2.2). Start attention (2.3) only if the profile ranks it. | Checks on the weakest load. Scored run under the same rule. |
| Nov 18 | Feature freeze. Three repeats on different traces, cold restarts, patch trim, pull-request hygiene. | Scored run: the safe best. |
| Nov 19 | Spare. Fixes only. | Scored run only for a clear local gain at no load's expense; submit before noon PT because of the queue. |
Until RULES.md says whether the board keeps your best or your latest scored run, treat every scored run as replacing the last one. The exact quota windows are in the ledger on Serving harness & runs.
Sources: official rules summary (llms.txt); chenwainuo/okc README (serving requirements, active patches); Paradigm announcement; georgxzh (per-shape SGLang µs); alygg77 (layer budgets); LMSYS megakernel post (step breakdown, sync costs, draft fusion); SGLang low-latency mode post; SGLang Muse Spark Open cookbook (Hopper specifics). Links on the overview and prior art.