Muse Spark Open Kernel Challenge: a plan to match or beat the leaders
Five H200 kernel problems plus the serving track. Specs, baselines, roofline ceilings, per-problem designs, ranked levers and dead ends, compiled from the official task specs and the public write-ups of top teams. Written 2026-11-14; revised the same day to put the serving track first, because the kernel boards are closed and only the serving board pays.
Serving-first revision (Nov 14): read this before the kernel pages
The rest of this page was written as a kernel-board plan. Those boards are frozen and carry no prize, so the plan is re-ordered here around the only score that still moves. Rules below come from the official summary at openkernelchallenge.dev/llms.txt (read Nov 14) unless tagged notes, which means a participant's repo and needs confirming against RULES.md from okc serve init. The Serving track page has the tiered patch plan, byte budget and schedule. Serving harness & runs has the local judge clone, the open questions, the pre-submit checklist and the run-budget ledger. Each kernel page now opens with a "Serving fit" section.
What is actually scored
| Rule | Detail | What it changes in this plan |
|---|---|---|
| Base and image | A patch on SGLang v0.5.23 (commit 297e54fd), run in lmsysorg/sglang:v0.5.23 on a whole 8×H200 node. | Not the kernel image (dev-muse-spark-open). Re-check that every kernel compiles in the v0.5.23 image with no network. |
| Score | At each load: your tokens/s per GPU at stock's per-user speed, divided by stock's. Final score is the geomean of those ratios; stock is 1.00. | Four loads weigh equally, so the weakest one caps the score. Inside a load, time adds up across 62 layers: rank work by µs saved per step, not by per-shape ratio. |
| Loads | 1, 4, 16 and 32 users. Scored runs cover all four; a check run takes one (16 by default). notes Hidden multi-turn agentic-coding conversations, exactly 220 output tokens per request. | Concurrency 4 has no kernel-board shape. Measure it; don't interpolate. |
| Gates | Lossless: no lower precision, approximation or skipped work beyond stock. TTFT at most 2× stock at each load. Build memory at most stock + 2 GB per GPU. notes Outputs must agree with stock about as well as stock agrees with itself. | Several kernel-track allowances and weight re-layouts may not survive these gates. See "Kernel levers that change" below. |
| Patch scope | Only python/sglang source and JSON configs. Nothing under python/sglang/srt/model_loader/, no renames, no binaries. At most 400 KB, no added line over 400 characters. Launch flags go in the flags file, except those marked fixed. | Kernel-track files could be 1 MB each, and many ran past 10,000 lines with a build per shape. The whole patch now has a 400 KB budget: pick parts, don't port everything. |
| Review | SGLang maintainers judge whether the patch could be merged, and that judgment counts toward the winners. Special-casing the workload or the load levels, breaking other models or GPUs, or sprawl can disqualify. Top entries are rerun several times on held-out data. | Among close scores the cleaner patch wins. Dispatch on general ranges (token count, batch size), never on the benchmark's four loads. |
| Budget | 1 scored run and 3 check runs per day, reset at 00:00 UTC. Up to 90 minutes of node time per run; a check takes about 25. Kernel top 50 go first in the queue, then each problem's top 10, then the rest of the top 100. | From Nov 14 that is at most 7 scored runs and 21 checks (quota days Nov 14–19 UTC, plus the hours before the close at 07:59 UTC on Nov 20). Plan on 5 and 15. Every decision has to be made locally first. |
| Prizes | $50,000 plus a signed H200, then $30,000, $10,000, $3,000, $2,000, and $1,000 each for 6th–10th. The board stays private until the track ends. | The gap to #1 can't be read off a board. Estimate it from your own per-load ratios and the kernel-board spread. |
Why a kernel rank does not carry over
- Kernels alone won't separate the top. At close the decode board read 2.61× at #1, 2.50× at #10 and 2.40× at #50: fifty teams inside 9%, most on the same public lineage.
- Size and shape. Organizers report many kernel entries over 10,000 lines with a variant per tested shape. That fits neither 400 KB nor a maintainer review.
- The single-GPU judge left out what serving adds: cross-GPU communication, the scheduler, the draft model, prefill interleaved with decode, and host overhead. LMSYS counted about 1,450 kernel launches in one stock decode step (on GB300). Their megakernel fix covers batch 1 on Blackwell only, so on H200 that overhead is probably still there to take.
- Stock is weakest on Hopper experts. SGLang's cookbook falls back to Marlin W4A16 on H200 and has no MegaMoE or sequence-parallel prefill there. That is where a patch has the most room: Marlin alone is about half the stock layer at one request (141.9 of 266.9 µs at verify b1) and more at 16 and 32.
Kernel levers that change under serving rules verify each
| Lever on this site | Status in serving |
|---|---|
| Per-shape builds and dispatch | Legal on the kernel judge. In serving, a build per benchmark load reads as special-casing, and each build spends patch bytes. Keep a few builds at most, selected by a general token-count range, as stock's tile selection does. |
| Lossless weight re-encoding (12-bit BF16, 6-bit scales, KPERM re-layouts) | Still lossless, but a second copy of the weights would blow the +2 GB per GPU build-memory cap, and model_loader/ can't be touched. Re-encode in place after load, free the original, and measure peak memory during the build. |
| FP16 partial sums, approximate exp2 and rsqrt, FP8-stored P (allowed on the kernel boards from Nov 3) | The serving rule is "nothing below stock". Assume these are out unless stock's own path does the same or RULES.md says otherwise. This hits the attention designs hardest. |
| Cold-L2 and 4 µs-subtraction tuning (cold-code hygiene, pad sweeps, weighting of tiny shapes) | Judge artifacts. Serving has no forced flush between calls and no subtraction; cache state is whatever the previous layer left. Re-measure inside a real 62-layer step before keeping any of it. |
| PDL chaining, fewer launches, no cross-SM barriers | Transfers. Launch count and gaps are a larger share of the step at 1 and 4 users. |
| Per-concurrency DFlash tuning (on the serving page) | Two risks: the speculative flags may be marked fixed, and a policy keyed to the four loads is special-casing. Check the flags file first. |
Serving plan for the days left
| # | Step | Exit criterion |
|---|---|---|
| 1 | Pin the rules. Run okc serve init; read RULES.md, the flags file (which flags are fixed) and the serving API. Resolve every notes and verify item on this page. | A one-page contract: gates, fixed flags, what "lossless" allows. |
| 2 | Build a local copy of the serving judge (how). Stock v0.5.23 on a rented 8×H200 node. Replay multi-turn coding conversations with 220-token outputs at 1, 4, 16 and 32 users; record tokens/s per GPU, per-user speed and TTFT. Alternate stock and patched runs. | Stock-vs-stock repeats agree within about 2%. Anything smaller is noise. |
| 3 | Profile stock at each load before porting anything. Split one step into host and scheduler, draft, verify (experts, attention, projections), communication and prefill. Use a provider that exposes GPU performance counters: Nebius, AWS and Google Cloud do; RunPod and Vast.ai block them. | A µs-per-step table per load. It replaces the kernel-board shares as the ranking. |
| 4 | Port by share of serving time, inside a byte budget. Expect experts first, then whatever the profile ranks next. Give each part a size allowance out of 400 KB and one general code path. | Each part passes the agreement gate, beats stock outside noise at every load it touches, and regresses none. |
| 5 | Try the free knobs. Sweep the launch flags that are not fixed (scheduler, chunked prefill, cache) on the local rig. If per-user speed is measured on decode only (confirm), TTFT under 2× earns nothing extra, so headroom there can be traded for decode throughput, as a general policy only. | Best flag set on the local rig, with TTFT at or below about 1.6× stock. |
| 6 | Ration official runs. Checks go to the load the local rig predicts worst. The daily scored run goes only to a stack that already passed locally. Submit the safest best version a day early; treat the last window as a spare, not as the plan. | A scored entry above 1.00 at all four loads by Nov 17, then only low-risk additions. |
| 7 | Keep it shaped like a pull request. Small diff, general dispatch, tests, no dead variants, attribution headers on the Apache-2.0 code you build on. | A maintainer could review it in one sitting. |
Other ways to close the gap to first
- Cut the cost that does not grow with users. If the score is read along your own speed-versus-throughput curve, a 1.2× faster step scores about 1.9, because the load-1 ratio alone is near 4. The same 1.2× on per-user cost only scores 1.2. Details and caveats: how the score rewards a faster step.
- Run the experiments the judge could not measure. Six are listed on the serving page, starting with token counts between the board sizes.
- Fix the weakest load first. With a geomean over four loads, lifting a 1.1× load to 1.3× is worth as much as lifting 2.0× to 2.36×. Loads 1 and 4 probably hinge on launch count and host overhead; 16 and 32 on expert and attention throughput and on how prefill interleaves with decode.
- Treat cached-context turns as the common case. The prefill problem was specified as 753 new tokens on an 80K-token conversation. If the hidden workload looks like that, prefix-cache hits and prefill scheduling matter more than raw prefill speed. This is an inference, not a published fact.
- Build on public Apache-2.0 kernels, with attribution. One top-150 team (yesthatwilly) credits public bases for nearly all of its late gains, with its own additions worth 0.1–2% per row. trxphxle's analysis puts the closing leader's edge in a different expert-kernel design, which is too large to rebuild in five days.
- Skip fusions that remove overlap. trxphxle measured a 4.2% layer loss from fusing the router GEMM with top-k and 2–4% from fusing the pre-expert norm, because the separate kernels were already overlapping.
- Aim for a repeatable score. Top entries are rerun several times on held-out data, so a robust 1.40× beats a lucky 1.45× that depends on one trace or one memory layout.
- Fix the loop, not only the agents. A finisher 5% off the kernel lead used a fresh worker per attempt and one integrator to merge wins; long-lived agents stalled and repeated narrow experiments. Keep the harness human-owned and frozen.
- Count mergeability as score. If serving scores cluster the way kernel scores did, maintainer judgment may decide the order among near-ties.
Kernel boards (closed): scoreboard and reference targets
Each score is the judge's geomean speedup over SGLang's own kernel on one H200, timed with cold L2 and CUDA-graph replay. "Best public" is the strongest open-source file set (keell0renz, Apache-2.0, board snapshot of Oct 31). "Ceiling" is my roofline estimate at a measured ~4.3 TB/s HBM rate plus realistic fixed costs. "Target" is the per-shape plan on each page.
| Problem | What it times | Best public | Closing #1 | Ceiling (est.) | Plan target |
|---|---|---|---|---|---|
| muse_attention_decode | Gated GQA attention, FP8 paged KV, ~80K context, decode and 8-token verify | 2.717× | 3.339× | 3.3–3.6× | ≈3.45× |
| muse_moe_w4a16 | Top-8 of 256 experts (32 local), NVFP4 W × BF16 A, SwiGLU, per-expert norm | 2.420× | 2.844× | 3.1–3.6× | ≈3.2× |
| muse_decode_layer | Whole decoder layer (sets the overall rank) | 2.266× | 2.614× | 2.9–3.1× | ≈2.85× |
| muse_bf16_proj | 5 dense projections plus the FP32 router, m = 1…256 | 1.588× | 1.728× | ≈1.9× | ≈1.78× |
| muse_prefill_layer | Whole layer at 753, 3,012 and 16,384-token prefills | 1.913× | 2.147× | >2.5× (FP8 attention) | ≈2.3× |
These boards are closed, so the targets are reference points, not goals. The closing spread shows how little kernels separate the top. #1 / #10 / #50 read 2.61 / 2.50 / 2.40× on decode, 3.34 / 3.01 / 2.81× on attention, 2.84 / 2.68 / 2.53× on MoE, 1.73 / 1.67 / 1.60× on projections and 2.15 / 2.04 / 1.95× on prefill (Quesma's snapshot at close, pending organizer review).
Leaders at close, from public write-ups: attention 3.339× (pjmullin3); decode-layer #1 was 2.4178× (mpeng19) on Oct 31 and 2.614× at close. Judge noise is about ±1.5% (1σ ≈ 0.63% per run), and the board keeps each team's best draw, so a row reads about 1σ high.
Kernel strategy in one paragraph
Don't start from scratch. Start from the strongest public lineage (vjeux → alygg77 → keell0renz, all Apache-2.0, with attributions kept). Reproduce its judge numbers in a local harness that copies the judge (cold L2, graph replay, AB/BA pairing). Then spend effort only on structural changes: tweaks to a starting kernel bought low single digits, while the big steps came from weight re-encoding, per-batch-size builds and the stream-K MoE. The gap between best public and #1 is 4% (bf16) to 23% (attention). The ceiling analysis puts most of it in five places:
- Attention, full-attention verify at batch 16/32. Most teams sit at 1.0–1.2× over FA3 here. With 8 tokens × 8 query heads per KV head, the tile is exactly 64 rows for an FP8 wgmma. → attention
- Attention, sliding-window latency at batch 1. 46-weight shapes at 4–9 µs where three dependent DRAM reads (length → page table → keys) and cold instruction fetch dominate.
- MoE fixed costs. A ~3 µs ramp, the A→B handoff dip and a 6–9 µs tail at small batch, plus split-K atomics that cost up to ~77 µs at 256 tokens. → MoE
- Decode layer: it's mostly MoE. alygg77's budgets put MoE at 84 of 154 µs (d32s) and 188 of 306 µs (v32s), with dense GEMVs ~16 µs over floor and <1.1 µs idle between kernels. Port the best MoE first, then the GEMVs. My earlier prefetch and reorder ideas were measured as losses. → decode layer
- Prefill: glue and the expert kernel. About 1.1 ms of memory-bound glue (13.5% of the load) and an instruction-bound expert kernel at T=753 (423 TFLOP/s, ~25% padding). FP8 attention helps only the full shapes, because attention is just 12.3% of the first-turn load. → prefill
Cross-cutting levers (kernel judge)
Written for the kernel judge. Several rows change under serving rules: read "Kernel levers that change under serving rules" above before porting any of them.
| Lever | Why it matters on this judge | EV |
|---|---|---|
One launch per call wherever possible; otherwise a PDL chain with griddepcontrol.launch_dependents issued right after the producer's last weight load | The judge subtracts only ~4 µs of graph overhead. Every extra launch and grid drain is paid in full, and 5 µs shapes move the geomean as much as 300 µs ones. The PDL trigger alone measured +0.9% per layer. | High |
| Per-shape builds and dispatch (branching on shape and config is legal) | Each batch size is timed separately. Specialized MoE builds per M bought several % per shape. | High |
| Lossless weight re-encoding: 12-bit BF16 (sign + mantissa + 4-bit exponent index, exact escapes), 6-bit NVFP4 block-scale indices | Small-m dense and MoE work is weight-bandwidth bound. The pooler at m=1 runs at 13.0 µs, under its 14.0 µs BF16 byte floor. Each NVFP4 tensor has only ~33 distinct E4M3 scale bytes. | High |
| Cold-code hygiene: keep prologue and epilogue code small, use rolled loops, warm the merge path early (dry run with stores off) | The L2 flush also evicts code. One team saw an attention merge take ~6,900 cycles cold vs ~2,300 warm. | Medium |
Pin every scalar load up front (empty asm volatile to stop nvcc sinking loads) | The compiler sank a scale load to the kernel tail, putting a cold DRAM read on the critical path. | Medium |
| No cross-SM barriers on the critical path; use data flags or counters per dependency | A 132-CTA flag hop costs 1.4 µs idle and 6.3 µs under a 4.3 TB/s stream. Fences cost 3.6–4.0 µs with TMA loads in flight. | High |
| Self-resetting counters for last-CTA reductions (no memset nodes) | torch.zeros in capture becomes an extra graph node on every replay. | Medium |
Read PDL-produced data with ld.global.cg / __ldcg | Correctness: L1 may hold the previous replay's data. | Must |
Dead ends already paid for by others (don't repeat)
- Whole-layer persistent megakernel on the single-GPU judge. Cross-SM hop cost under HBM load kept it below chain + PDL (keell0renz). The LMSYS megakernel wins on 8×GB300 at batch 1, where it also removes communication; see serving.
- Fused-slice MoE (A and B per I-slice in one CTA): six variants, all 5–13% slower. Dual-lane wgmma MoE and per-unit barriers: 8–20% slower.
- L2 prefetch of next-layer weights during attention: made the layer slower, because extra DRAM traffic stretched attention's dependent round trips. Expert-weight L2 prefetch during the prior tail: ×0.99–1.00.
- FA4-style polynomial exp2 everywhere: −12–21% where exp wasn't the bottleneck. Use it only on the shape where the MUFU count is binding (full verify, batch 32).
- Tensor-core GEMV for small-m BF16: 0.2–2.5 TB/s vs 2.6–3.7 TB/s for CUDA-core and TMA-ring kernels.
- Approximation tricks (attending to only the last 2–16K keys, dropping low-weight experts): both fail the router check and the residual bound, and both are banned anyway.
Kernel build order (reference; superseded by the serving plan above)
| Phase | Work | Exit criterion |
|---|---|---|
| 0. Harness (½ day) | Build the judge clone (methodology): 1 GiB L2 flush, graph replay, AB/BA paired medians, 4 µs subtraction, load weights, the router check. JIT cache on CPU. Per-SM %globaltimer timelines. | Public keell0renz files reproduce their judge numbers within 2%. |
| 1. Baseline (½ day) | Adopt keell0renz files per problem. Fix the known prefill fresh-thread mis-route (bind the CUDA context per thread). | All 5 pass the pre-timing tests; scores are within noise of public. |
| 2. Attention (2–3 days) | Full-verify FP8 wgmma kernel with two KV-head warpgroups decoupled. Full decode as balanced persistent split-KV. Sliding-path prologue diet. | v16/v32 full ≥2.6× FA3; geomean ≥3.3×. |
| 3. MoE (2–3 days) | Swap-AB with dequant into wgmma A registers, stream-K without atomics, B-weight prefetch before the A→B flag, tail LPT ordering. | ≥2.9×, with no regression at M=1. |
| 4. bf16 (1 day) | K-stacked hi/lo router GEMM, cluster split-K for qkv, stream-K wgmma for m=128/256; evaluate an entropy-coded exponent (~11 bit). | ≥1.73×. |
| 5. Decode layer (2 days) | Port the new attention and MoE. Router-first reorder. Weight-prefetch-before-wait only behind latency-bound ops. | ≥2.75×. |
| 6. Prefill (2 days) | FP8 QKᵀ attention with GQA-packed 128-row q-tiles. Fused QKV epilogue (norm, RoPE, cache write). Pooler and router in one GEMM. Persistent grouped MoE. | ≥2.2×. |
| Serving (now → Nov 19) | Superseded by the serving plan at the top of this page. Two items from the old row need a rules check before any work: per-concurrency DFlash tuning (fixed flags, load-level special-casing) and the fused all-reduce (patch scope and mergeability). | Beats stock throughput at stock interactivity at all 4 loads. |
Model cheat sheet (one H200 under TP8/EP8)
| Item | Value |
|---|---|
| Model | 816B MoE, ~42.7B active, 62 decoder layers (46 sliding-window at 2,048 tokens with RoPE; 16 full-attention, no RoPE) |
| Hidden | 8192, FP32 residual stream s, BF16 activations |
| Attention per GPU | 16 query heads, 2 KV heads, head_dim 64. q is RMSNormed and scaled by 5.473; logits /8. Output = RMSNorm(attn) × sigmoid(gate). FP8 E4M3 paged cache, page size 1 in scored shapes. |
| Projections | qkv_gate [2304×8192]; o [8192×1024]; pooler [4096×8192]; unpooler (only the first 1024 used at T≤8); router FP32 [256×8192] |
| Router | top8(sigmoid(0.177·h·Wᵀ) + bias). Picks must score within 0.05 of the reference's 8th; clear winners (margin >0.03, or >0.04 above 256 tokens) must be picked. |
| Experts | 256 total, 32 local (EP8), width 4096, NVFP4 (E2M1 codes + E4M3 per-16 scales), SwiGLU, RMSNorm on each expert output × (1+post_norm_w), weighted sum. Routing is skewed; each test makes this GPU the busiest. |
| Speculative decoding | DFlash V2. Verify shapes use 8-token blocks (q_len 8). |
RULES.md before it drives work.Sources
- Paradigm announcement · openkernelchallenge.dev
- Task specs mirrored in lilaclilac09/open-kernel-challenge (TASK.md per problem)
- keell0renz (best public, plus LESSONS.md) · alygg77 · vjeux
- yesthatwilly (closing-board leaders) · georgxzh (per-shape SGLang times, rule timeline) · Kushagrabainsla · infraax · maximenewman · lee101 · trxphxle · shashank-100
- LMSYS: Muse Spark Open megakernel · chenwainuo (serving notes)
- Serving-first revision: official rules summary (llms.txt) · Quesma: reports from the top 50 · SGLang Muse Spark Open cookbook