Serving track: where the prizes are

Opened Nov 12, 9 am PT; closes Nov 19, 11:59 pm PT. Private leaderboard. A patch to SGLang v0.5.23 serving Muse Spark Open on one 8×H200 node. Revised Nov 14 against the official rules summary.

What changed in this revision. The earlier version of this page assumed the kernel work would transfer almost one-to-one and ranked eleven patches by kernel-board speedup. The current rules change that. Patches must be lossless, fit in 400 KB of python/sglang source, stay within 2 GB per GPU of stock build memory, and pass a maintainer review that can disqualify load-specific code. About seven scored runs remain. So the plan is now tiered: free and small general patches first, to get a safe score on the board, then kernel ports one at a time, each with a byte allowance. Per-concurrency DFlash tuning, the fused all-reduce, expert re-sharding and the megakernel have moved down or out.

What is scored

From the official summary (llms.txt, read Nov 14) unless tagged notes, which means a participant's repo: confirm against RULES.md from okc serve init. The full table is on the overview.

How the kernel work maps onto serving

MapsDoes not map
  • With DFlash on, every decode step is an 8-token verify. 1, 16 and 32 users are the kernel boards' verify b1/b16/b32 shapes: 8, 128 and 256 tokens.
  • 4 users is 32 tokens. The expert and projection kernels have an m=32 build; no attention board shape matches.
  • Each step is 62 layers (46 sliding, 16 full) of the decode layer.
  • Communication is no longer identity. TP8/EP8 exchanges sit between kernels, and the step waits for the slowest GPU.
  • No forced L2 flush and no 4 µs subtraction. Tuning aimed at those judge mechanics has to be re-measured.
  • The kernel-track numeric allowances (FP16 partial sums, approximate exp2, FP8-stored P) are not in the serving rules.
  • One build per board shape and 1 MB files fit neither 400 KB nor a review.
  • The draft model, scheduler, sampling, prefill interleave and host overhead were never on a kernel board.

Where the stock step goes, from the kernel judge's own numbers

SGLang's per-layer times on the judge (georgxzh's run, tabulated on the decode-layer and MoE pages) give a first estimate of stock layer-kernel time per verify step: 46 sliding layers plus 16 full layers.

Users (verify tokens)Sliding / full layer (µs)Layer kernels per stepExtra from the 16 full-attention layersStock experts alone, per layer
1 (8)266.9 / 270.416.6 ms0.06 ms (0.3%)141.9 µs
16 (128)523.5 / 694.435.2 ms2.7 ms (8%)459.2 µs
32 (256)732.0 / 1,086.851.1 ms5.7 ms (11%)617.0 µs

Step-time model

request(c)  ≈ TTFT(c) + (220 / accepted_tokens_per_step) × step(c)
TTFT(c)     ≈ queue wait + prefill(new tokens, given the cached prefix)
step(c)     ≈ draft(c) + Σ_62 layers [ layer_kernels(c, q_len=8) + comm(c) ] + host/scheduler
per-user speed(c) ≈ accepted_tokens_per_step / step(c)        # confirm how the judge defines it
tokens/s per GPU  ≈ c × 220 / request(c) / 8
# the score holds per-user speed at stock's value, so a faster step is cashed in as more users served

Measure the breakdown before porting anything: one trace of a step at 1 user and one at 32, CUDA graphs on, split into (a) layer kernels by family, (b) communication, (c) draft, (d) host and gaps, (e) prefill's share of GPU time. Two unknowns decide the plan. The first is communication on stock H200: LMSYS's GB300 megakernel layout has about five sync stops per layer, and stock collectives may cost far more than its 2 µs flags. The second is launch count: LMSYS counted about 1,450 kernel launches per stock step on GB300, and 12.9 ms falling to 5.9 ms with a megakernel at one request.

Patch plan, in tiers

The order is by expected gain per unit of risk and bytes, not by kernel-board speedup. Each row needs a local A/B outside noise at every load it touches before it joins the patch.

Tier 0: no code

#ChangeWhyEVRisk
0.1Sweep the launch flags that are not marked fixed: scheduler policy, chunked-prefill size, running-request cap, cache settings.Free, zero patch bytes, and it shows how much of the frontier is configuration. If per-user speed is measured on decode only, TTFT below 2× stock earns nothing, so headroom there can be traded for decode throughput.Unknown until measuredLow. Keep TTFT at or below about 1.6× stock for noise.

Tier 1: small general patches (the first scored entry)

#PatchWhy / evidenceEVRule and merge risk
1.1Marlin tile and thread selection on Hopper, by token-count range.chenwainuo's active serving patches cap the expert tile at 16 through 256 tokens and 32 through 512, and leave larger prefills on stock selection. No numbers published, but it is a few lines on the largest cost in the layer.MediumLow. A range rule is general; a rule per benchmark load is not.
1.2Fused finalization at small batch for the expert output.Also in chenwainuo's active set; their stage list ends with a finalizer after the down GEMM. Fewer launches at 1 and 4 users.MediumLow. Keep stock's BF16 rounding points.
1.3PDL chaining inside the CUDA graph: programmatic stream serialization, with producers triggering after their last weight issue.+0.9% per layer measured in isolation on the kernel judge. SGLang's own low-latency work launches dependent kernels early the same way.MediumLow.
1.4Host path: no Python on the step path, overlap scheduler on, graphs captured for the batch sizes that occur.220-token requests at one user are latency-sensitive end to end.MediumEach captured graph holds memory: watch the +2 GB per GPU cap.
1.5Prefill and decode interleave: a general policy for how much prefill may run between decode steps, plus prefix-cache reuse across turns.Multi-turn conversations share long prefixes. At 16 and 32 users a new turn's prefill stalls everyone's decode.High at 16 and 32Must be a general policy with a sane default, not a setting per load.

Tier 2: kernel ports, one at a time

#PortWhy / evidenceEVRule and merge risk
2.1Experts: the persistent NVFP4 kernel in place of Marlin at decode and verify token counts, under SGLang's fused-MoE interface (MoE page).The largest term in the stock layer (table above). 2.42× best public, 2.84× at the board's close. Stock has no MegaMoE on Hopper.HighRe-layout in place after load: no second copy, nothing in model_loader/. Two or three builds by token range, not one per board shape. FP32 partial sums as in stock.
2.2Dense projections: lossless 12-bit GEMVs at 32 tokens and below; stock path above (bf16 page).Dense GEMVs were 51.5 of 154 µs in an optimized layer against a 34.9 µs floor (alygg77): about 16 µs a layer, 1 ms a step.High at 1 and 4The memory cap allows about 32 MB of duplicated weights per layer, so both encodings of every matrix will not fit. Converting in place slows prefill GEMMs: check TTFT.
2.3Attention: full-attention decode and verify only, as a backend for this model (attention page).Worth up to about 5.7 ms of a 51 ms step at 32 users with long contexts; near nothing at one user.Medium, conditional on the profileThe highest rule risk: the FP16 and approximate-math paths are kernel-track allowances. Port only the scheduling and gather structure unless RULES.md allows more.

Tier 3: check first, or leave out

ItemEarlier rankNow
Fused all-reduce + residual + RMSNorm#4, HighCheck whether the v0.5.23 image already ships an all-reduce fusion that works on Hopper for this model. If it is a switch in Python, it becomes Tier 1. If it needs new cross-process GPU code, leave it: the bytes, hang risk and review cost are too high for five days.
DFlash tuning per concurrency#6, HighThe speculative flags may be fixed, a policy keyed to the four loads is special-casing, and block verification changes the sampling path that the agreement gate checks. At most one general setting, and only if the flags file leaves it open.
Draft model as fused launches#9, MediumOnly if the profile puts the draft above about a tenth of the step. On GB300 it was 1.16 of 12.9 ms.
Expert re-sharding (N-sharded TP)#8, MediumOut. It changes the model's parallel layout and the loader's job, and it is too large to review.
Per-layer megakernel#11, LowOut. LMSYS's version is batch 1 and Blackwell only, and far too large for 400 KB.

How the score rewards a faster step

Read literally, the rule compares throughput at the same per-user speed. A faster system reaches stock's speed with more users on board, so the ratio for a load is roughly "how many users can I carry at the speed stock gave that many". That makes the four ratios very unequal. A sketch, with stock step time taken as 15.5 ms plus 1.1 ms per user (a line through the judge-derived 16.6, 35.2 and 51.1 ms above):

Change to the stepLoad 1Load 4Load 16Load 32Geomean
Everything 1.2× faster3.991.901.371.291.91
Only the fixed part 1.2× faster3.331.581.151.071.59
Only the per-user part 1.2× faster1.201.201.201.201.20
Everything 2× faster15.965.492.872.444.98

Kernel experiments the public ledgers do not cover

Checked against keell0renz's lessons file and alygg77's negative-results and measured-facts files. Every ledger was built on the single-GPU judge, so the gaps are where serving differs from it. All are hypotheses.

#ExperimentWhy it is openFirst testOdds
1Token counts between the board sizesThe public expert kernels were built per scored size (1, 8, 16, 32+) and the judge timed discrete sizes. In serving the running batch moves, because requests in prefill are not decoding. A dedicated tile for 17–32 tokens ran 2.2–3.0 µs slower at d32, so that range is sensitive.Sweep every multiple of 8 from 8 to 256 on the expert kernel and the dense dispatch. Plot µs against tokens and remove the cliffs.High that it matters
2KV page size and contiguous readsThe boards fixed page size 1 with arbitrary rows, so every key was a 128-byte gather. The only layout experiment in the ledgers was sorting pages inside the kernel, which lost 162–276 µs per full load. Page size is a launch flag: SGLang's guide pairs 64 with long contexts and 1 with maximum multi-turn reuse.If the flag is open: a bare read skeleton, rows against 16- or 64-token pages, at 32 requests and 80K context. Then a bulk-read path. Cost: coarser prefix matching, roughly up to 63 tokens re-prefilled per turn.Medium. Full layers at 16 and 32 users only.
3Draft model and sampling pathNo board covered them. On GB300 the draft was 1.16 ms of a 12.9 ms step, and 8 launches in place of 175 kernels took it to 0.44 ms.Profile the draft's share at 1 and 32 users. Check whether a general dense GEMV port already covers the draft's projections.Medium at 1 and 4 users
4Evict-first hints on weight streamsLMSYS's megakernel tags weight streams so they do not push small hot data out of L2. The ledgers only say "twin L2 policies" were at optimum or worse, with no numbers, and under the judge's flush.Tag the bulk weight copies and measure step time inside serving, not on a flushed harness.Low, but a few lines
5The unexplained memory-layout effectAdding 3 MB of scratch cost 0.5–1.3 µs at d1 and v1 with identical machine code, and the mechanism was never measured. Each GPU streams 10 GB or more of weights per step, so a per-allocation cost would recur in all 62 layers.One arena with fixed alignment for all re-encoded weights against default allocation, over at least four layouts.Unknown
6Toolchain and compile timeThe best public kernels compile native CUDA at start-up, and the serving image is not the kernel image.In the v0.5.23 image: does that compile path exist, how long do eight ranks take, and can the port ride the JIT route stock already uses?Blocking. Do it first.

Tempting, but skip. Prefetching weights into L2 during communication waits or inside one persistent kernel: with phase A fully L2-resident the expert kernel gained only 0.4–0.65 µs of 22. Prefetching experts predicted from the previous step's routes: LMSYS gained about 0.2 ms of a 5.3 ms verify, then found the prefetch was evicting code and ran faster with it off, on a GPU with a larger L2. Keeping old K/V rows warm: 0.2–0.3 µs at 8 tokens or fewer.

Byte budget

Planning allowances, not measurements. Strip dead variants, comments and per-shape builds first, then measure each part's real diff size.

PartAllowanceNote
Experts (2.1)160 KBThe part most worth its bytes.
Dense projections (2.2)80 KBOne GEMV kernel plus the encoder.
Attention (2.3)90 KBSpend only if the profile and the rules both allow it.
Tier 1, glue, tests40 KBTests count toward mergeability.
Reserve30 KBStay under about 390,000 bytes in case KB means 1,000.

Gates before every submission

The full checklist and commands are on Serving harness & runs. In short:

Schedule, Nov 14 → 19, with the run budget

Day (PT)Local workOfficial runs
Nov 14Pin the rules (okc serve init, RULES.md, fixed flags). Bring up stock v0.5.23 on 8×H200. Stock baselines at 1/4/16/32, twice. Profile a step at 1 and 32 users.One check at 16 users with a flags-only or no-op patch, to learn what a run reports.
Nov 15Tier 0 sweep. Tier 1 patches with the agreement gate.Checks at 1 and 32 users. First scored run once all four loads beat stock locally.
Nov 16Experts (2.1) behind a general range dispatch. Memory audit.Checks at 16 and 1. Scored run only if it beats the current entry locally at every load.
Nov 17Dense projections (2.2). Start attention (2.3) only if the profile ranks it.Checks on the weakest load. Scored run under the same rule.
Nov 18Feature freeze. Three repeats on different traces, cold restarts, patch trim, pull-request hygiene.Scored run: the safe best.
Nov 19Spare. Fixes only.Scored run only for a clear local gain at no load's expense; submit before noon PT because of the queue.

Until RULES.md says whether the board keeps your best or your latest scored run, treat every scored run as replacing the last one. The exact quota windows are in the ledger on Serving harness & runs.

Sources: official rules summary (llms.txt); chenwainuo/okc README (serving requirements, active patches); Paradigm announcement; georgxzh (per-shape SGLang µs); alygg77 (layer budgets); LMSYS megakernel post (step breakdown, sync costs, draft fusion); SGLang low-latency mode post; SGLang Muse Spark Open cookbook (Hopper specifics). Links on the overview and prior art.