Muse Spark Open Kernel Challenge: a plan to match or beat the leaders

Five H200 kernel problems plus the serving track. Specs, baselines, roofline ceilings, per-problem designs, ranked levers and dead ends, compiled from the official task specs and the public write-ups of top teams. Written 2026-11-14; revised the same day to put the serving track first, because the kernel boards are closed and only the serving board pays.

Where the challenge stands today (Nov 14). The kernel track closed Nov 4, 23:59 PT; the boards froze Nov 5, 07:59 UTC, and final ranks come from organizer reruns on held-out shapes. The serving track, where the $100K is paid, opened Nov 12, 9 am PT and closes Nov 19, 11:59 pm PT (it slipped from Nov 5 to Nov 10, then to Nov 12). Its leaderboard is private: each team sees only its own score. Since Nov 11, patches build on SGLang v0.5.23 and must be lossless. A kernel can no longer move a kernel board. What it can still do is make SGLang faster in a serving submission, because the kernel loads (batch 1/16/32, decode and 8-token verify) cover three of the serving track's concurrency levels. The fourth is 4 users, which no kernel shape covers. The kernel pages are now a parts catalogue rather than the plan: a serving patch is capped at 400 KB, goes through a maintainer review, and may not drop below stock precision. Start with the serving-first section below, then see Serving track.

Serving-first revision (Nov 14): read this before the kernel pages

The rest of this page was written as a kernel-board plan. Those boards are frozen and carry no prize, so the plan is re-ordered here around the only score that still moves. Rules below come from the official summary at openkernelchallenge.dev/llms.txt (read Nov 14) unless tagged notes, which means a participant's repo and needs confirming against RULES.md from okc serve init. The Serving track page has the tiered patch plan, byte budget and schedule. Serving harness & runs has the local judge clone, the open questions, the pre-submit checklist and the run-budget ledger. Each kernel page now opens with a "Serving fit" section.

What is actually scored

RuleDetailWhat it changes in this plan
Base and imageA patch on SGLang v0.5.23 (commit 297e54fd), run in lmsysorg/sglang:v0.5.23 on a whole 8×H200 node.Not the kernel image (dev-muse-spark-open). Re-check that every kernel compiles in the v0.5.23 image with no network.
ScoreAt each load: your tokens/s per GPU at stock's per-user speed, divided by stock's. Final score is the geomean of those ratios; stock is 1.00.Four loads weigh equally, so the weakest one caps the score. Inside a load, time adds up across 62 layers: rank work by µs saved per step, not by per-shape ratio.
Loads1, 4, 16 and 32 users. Scored runs cover all four; a check run takes one (16 by default). notes Hidden multi-turn agentic-coding conversations, exactly 220 output tokens per request.Concurrency 4 has no kernel-board shape. Measure it; don't interpolate.
GatesLossless: no lower precision, approximation or skipped work beyond stock. TTFT at most 2× stock at each load. Build memory at most stock + 2 GB per GPU. notes Outputs must agree with stock about as well as stock agrees with itself.Several kernel-track allowances and weight re-layouts may not survive these gates. See "Kernel levers that change" below.
Patch scopeOnly python/sglang source and JSON configs. Nothing under python/sglang/srt/model_loader/, no renames, no binaries. At most 400 KB, no added line over 400 characters. Launch flags go in the flags file, except those marked fixed.Kernel-track files could be 1 MB each, and many ran past 10,000 lines with a build per shape. The whole patch now has a 400 KB budget: pick parts, don't port everything.
ReviewSGLang maintainers judge whether the patch could be merged, and that judgment counts toward the winners. Special-casing the workload or the load levels, breaking other models or GPUs, or sprawl can disqualify. Top entries are rerun several times on held-out data.Among close scores the cleaner patch wins. Dispatch on general ranges (token count, batch size), never on the benchmark's four loads.
Budget1 scored run and 3 check runs per day, reset at 00:00 UTC. Up to 90 minutes of node time per run; a check takes about 25. Kernel top 50 go first in the queue, then each problem's top 10, then the rest of the top 100.From Nov 14 that is at most 7 scored runs and 21 checks (quota days Nov 14–19 UTC, plus the hours before the close at 07:59 UTC on Nov 20). Plan on 5 and 15. Every decision has to be made locally first.
Prizes$50,000 plus a signed H200, then $30,000, $10,000, $3,000, $2,000, and $1,000 each for 6th–10th. The board stays private until the track ends.The gap to #1 can't be read off a board. Estimate it from your own per-load ratios and the kernel-board spread.

Why a kernel rank does not carry over

Kernel levers that change under serving rules verify each

Lever on this siteStatus in serving
Per-shape builds and dispatchLegal on the kernel judge. In serving, a build per benchmark load reads as special-casing, and each build spends patch bytes. Keep a few builds at most, selected by a general token-count range, as stock's tile selection does.
Lossless weight re-encoding (12-bit BF16, 6-bit scales, KPERM re-layouts)Still lossless, but a second copy of the weights would blow the +2 GB per GPU build-memory cap, and model_loader/ can't be touched. Re-encode in place after load, free the original, and measure peak memory during the build.
FP16 partial sums, approximate exp2 and rsqrt, FP8-stored P (allowed on the kernel boards from Nov 3)The serving rule is "nothing below stock". Assume these are out unless stock's own path does the same or RULES.md says otherwise. This hits the attention designs hardest.
Cold-L2 and 4 µs-subtraction tuning (cold-code hygiene, pad sweeps, weighting of tiny shapes)Judge artifacts. Serving has no forced flush between calls and no subtraction; cache state is whatever the previous layer left. Re-measure inside a real 62-layer step before keeping any of it.
PDL chaining, fewer launches, no cross-SM barriersTransfers. Launch count and gaps are a larger share of the step at 1 and 4 users.
Per-concurrency DFlash tuning (on the serving page)Two risks: the speculative flags may be marked fixed, and a policy keyed to the four loads is special-casing. Check the flags file first.

Serving plan for the days left

#StepExit criterion
1Pin the rules. Run okc serve init; read RULES.md, the flags file (which flags are fixed) and the serving API. Resolve every notes and verify item on this page.A one-page contract: gates, fixed flags, what "lossless" allows.
2Build a local copy of the serving judge (how). Stock v0.5.23 on a rented 8×H200 node. Replay multi-turn coding conversations with 220-token outputs at 1, 4, 16 and 32 users; record tokens/s per GPU, per-user speed and TTFT. Alternate stock and patched runs.Stock-vs-stock repeats agree within about 2%. Anything smaller is noise.
3Profile stock at each load before porting anything. Split one step into host and scheduler, draft, verify (experts, attention, projections), communication and prefill. Use a provider that exposes GPU performance counters: Nebius, AWS and Google Cloud do; RunPod and Vast.ai block them.A µs-per-step table per load. It replaces the kernel-board shares as the ranking.
4Port by share of serving time, inside a byte budget. Expect experts first, then whatever the profile ranks next. Give each part a size allowance out of 400 KB and one general code path.Each part passes the agreement gate, beats stock outside noise at every load it touches, and regresses none.
5Try the free knobs. Sweep the launch flags that are not fixed (scheduler, chunked prefill, cache) on the local rig. If per-user speed is measured on decode only (confirm), TTFT under 2× earns nothing extra, so headroom there can be traded for decode throughput, as a general policy only.Best flag set on the local rig, with TTFT at or below about 1.6× stock.
6Ration official runs. Checks go to the load the local rig predicts worst. The daily scored run goes only to a stack that already passed locally. Submit the safest best version a day early; treat the last window as a spare, not as the plan.A scored entry above 1.00 at all four loads by Nov 17, then only low-risk additions.
7Keep it shaped like a pull request. Small diff, general dispatch, tests, no dead variants, attribution headers on the Apache-2.0 code you build on.A maintainer could review it in one sitting.

Other ways to close the gap to first

Kernel boards (closed): scoreboard and reference targets

Each score is the judge's geomean speedup over SGLang's own kernel on one H200, timed with cold L2 and CUDA-graph replay. "Best public" is the strongest open-source file set (keell0renz, Apache-2.0, board snapshot of Oct 31). "Ceiling" is my roofline estimate at a measured ~4.3 TB/s HBM rate plus realistic fixed costs. "Target" is the per-shape plan on each page.

ProblemWhat it timesBest publicClosing #1Ceiling (est.)Plan target
muse_attention_decodeGated GQA attention, FP8 paged KV, ~80K context, decode and 8-token verify2.717×3.339×3.3–3.6×≈3.45×
muse_moe_w4a16Top-8 of 256 experts (32 local), NVFP4 W × BF16 A, SwiGLU, per-expert norm2.420×2.844×3.1–3.6×≈3.2×
muse_decode_layerWhole decoder layer (sets the overall rank)2.266×2.614×2.9–3.1×≈2.85×
muse_bf16_proj5 dense projections plus the FP32 router, m = 1…2561.588×1.728×≈1.9×≈1.78×
muse_prefill_layerWhole layer at 753, 3,012 and 16,384-token prefills1.913×2.147×>2.5× (FP8 attention)≈2.3×

These boards are closed, so the targets are reference points, not goals. The closing spread shows how little kernels separate the top. #1 / #10 / #50 read 2.61 / 2.50 / 2.40× on decode, 3.34 / 3.01 / 2.81× on attention, 2.84 / 2.68 / 2.53× on MoE, 1.73 / 1.67 / 1.60× on projections and 2.15 / 2.04 / 1.95× on prefill (Quesma's snapshot at close, pending organizer review).

Leaders at close, from public write-ups: attention 3.339× (pjmullin3); decode-layer #1 was 2.4178× (mpeng19) on Oct 31 and 2.614× at close. Judge noise is about ±1.5% (1σ ≈ 0.63% per run), and the board keeps each team's best draw, so a row reads about 1σ high.

Kernel strategy in one paragraph

Re-rank before using this. The five places below were chosen to move kernel-board geomeans, where a 5 µs shape counts as much as a 300 µs one. A serving step adds time across 62 layers, so what matters there is µs saved per step at each load. Expect experts and launch and host overhead to rise in rank and tiny-shape latency tricks to fall; let the stock profile (serving plan, step 3) decide.

Don't start from scratch. Start from the strongest public lineage (vjeux → alygg77 → keell0renz, all Apache-2.0, with attributions kept). Reproduce its judge numbers in a local harness that copies the judge (cold L2, graph replay, AB/BA pairing). Then spend effort only on structural changes: tweaks to a starting kernel bought low single digits, while the big steps came from weight re-encoding, per-batch-size builds and the stream-K MoE. The gap between best public and #1 is 4% (bf16) to 23% (attention). The ceiling analysis puts most of it in five places:

  1. Attention, full-attention verify at batch 16/32. Most teams sit at 1.0–1.2× over FA3 here. With 8 tokens × 8 query heads per KV head, the tile is exactly 64 rows for an FP8 wgmma. → attention
  2. Attention, sliding-window latency at batch 1. 46-weight shapes at 4–9 µs where three dependent DRAM reads (length → page table → keys) and cold instruction fetch dominate.
  3. MoE fixed costs. A ~3 µs ramp, the A→B handoff dip and a 6–9 µs tail at small batch, plus split-K atomics that cost up to ~77 µs at 256 tokens. → MoE
  4. Decode layer: it's mostly MoE. alygg77's budgets put MoE at 84 of 154 µs (d32s) and 188 of 306 µs (v32s), with dense GEMVs ~16 µs over floor and <1.1 µs idle between kernels. Port the best MoE first, then the GEMVs. My earlier prefetch and reorder ideas were measured as losses. → decode layer
  5. Prefill: glue and the expert kernel. About 1.1 ms of memory-bound glue (13.5% of the load) and an instruction-bound expert kernel at T=753 (423 TFLOP/s, ~25% padding). FP8 attention helps only the full shapes, because attention is just 12.3% of the first-turn load. → prefill

Cross-cutting levers (kernel judge)

Written for the kernel judge. Several rows change under serving rules: read "Kernel levers that change under serving rules" above before porting any of them.

LeverWhy it matters on this judgeEV
One launch per call wherever possible; otherwise a PDL chain with griddepcontrol.launch_dependents issued right after the producer's last weight loadThe judge subtracts only ~4 µs of graph overhead. Every extra launch and grid drain is paid in full, and 5 µs shapes move the geomean as much as 300 µs ones. The PDL trigger alone measured +0.9% per layer.High
Per-shape builds and dispatch (branching on shape and config is legal)Each batch size is timed separately. Specialized MoE builds per M bought several % per shape.High
Lossless weight re-encoding: 12-bit BF16 (sign + mantissa + 4-bit exponent index, exact escapes), 6-bit NVFP4 block-scale indicesSmall-m dense and MoE work is weight-bandwidth bound. The pooler at m=1 runs at 13.0 µs, under its 14.0 µs BF16 byte floor. Each NVFP4 tensor has only ~33 distinct E4M3 scale bytes.High
Cold-code hygiene: keep prologue and epilogue code small, use rolled loops, warm the merge path early (dry run with stores off)The L2 flush also evicts code. One team saw an attention merge take ~6,900 cycles cold vs ~2,300 warm.Medium
Pin every scalar load up front (empty asm volatile to stop nvcc sinking loads)The compiler sank a scale load to the kernel tail, putting a cold DRAM read on the critical path.Medium
No cross-SM barriers on the critical path; use data flags or counters per dependencyA 132-CTA flag hop costs 1.4 µs idle and 6.3 µs under a 4.3 TB/s stream. Fences cost 3.6–4.0 µs with TMA loads in flight.High
Self-resetting counters for last-CTA reductions (no memset nodes)torch.zeros in capture becomes an extra graph node on every replay.Medium
Read PDL-produced data with ld.global.cg / __ldcgCorrectness: L1 may hold the previous replay's data.Must

Dead ends already paid for by others (don't repeat)

Kernel build order (reference; superseded by the serving plan above)

Not a schedule any more. Phases 0–6 add up to roughly 10–13 days of work against boards that no longer move, with five days of serving left. Use the table as the order in which to pick parts once the serving profile says a part is worth its bytes.
PhaseWorkExit criterion
0. Harness (½ day)Build the judge clone (methodology): 1 GiB L2 flush, graph replay, AB/BA paired medians, 4 µs subtraction, load weights, the router check. JIT cache on CPU. Per-SM %globaltimer timelines.Public keell0renz files reproduce their judge numbers within 2%.
1. Baseline (½ day)Adopt keell0renz files per problem. Fix the known prefill fresh-thread mis-route (bind the CUDA context per thread).All 5 pass the pre-timing tests; scores are within noise of public.
2. Attention (2–3 days)Full-verify FP8 wgmma kernel with two KV-head warpgroups decoupled. Full decode as balanced persistent split-KV. Sliding-path prologue diet.v16/v32 full ≥2.6× FA3; geomean ≥3.3×.
3. MoE (2–3 days)Swap-AB with dequant into wgmma A registers, stream-K without atomics, B-weight prefetch before the A→B flag, tail LPT ordering.≥2.9×, with no regression at M=1.
4. bf16 (1 day)K-stacked hi/lo router GEMM, cluster split-K for qkv, stream-K wgmma for m=128/256; evaluate an entropy-coded exponent (~11 bit).≥1.73×.
5. Decode layer (2 days)Port the new attention and MoE. Router-first reorder. Weight-prefetch-before-wait only behind latency-bound ops.≥2.75×.
6. Prefill (2 days)FP8 QKᵀ attention with GQA-packed 128-row q-tiles. Fused QKV epilogue (norm, RoPE, cache write). Pooler and router in one GEMM. Persistent grouped MoE.≥2.2×.
Serving (now → Nov 19)Superseded by the serving plan at the top of this page. Two items from the old row need a rules check before any work: per-concurrency DFlash tuning (fixed flags, load-level special-casing) and the fused all-reduce (patch scope and mergeability).Beats stock throughput at stock interactivity at all 4 loads.

Model cheat sheet (one H200 under TP8/EP8)

ItemValue
Model816B MoE, ~42.7B active, 62 decoder layers (46 sliding-window at 2,048 tokens with RoPE; 16 full-attention, no RoPE)
Hidden8192, FP32 residual stream s, BF16 activations
Attention per GPU16 query heads, 2 KV heads, head_dim 64. q is RMSNormed and scaled by 5.473; logits /8. Output = RMSNorm(attn) × sigmoid(gate). FP8 E4M3 paged cache, page size 1 in scored shapes.
Projectionsqkv_gate [2304×8192]; o [8192×1024]; pooler [4096×8192]; unpooler (only the first 1024 used at T≤8); router FP32 [256×8192]
Routertop8(sigmoid(0.177·h·Wᵀ) + bias). Picks must score within 0.05 of the reference's 8th; clear winners (margin >0.03, or >0.04 above 256 tokens) must be picked.
Experts256 total, 32 local (EP8), width 4096, NVFP4 (E2M1 codes + E4M3 per-16 scales), SwiGLU, RMSNorm on each expert output × (1+post_norm_w), weighted sum. Routing is skewed; each test makes this GPU the busiest.
Speculative decodingDFlash V2. Verify shapes use 8-token blocks (q_len 8).
Honesty note. I had no H200 for this plan. The designs, targets and ceilings are engineering estimates from the published specs, judge numbers and write-ups. Every lever is marked with its evidence: measured by someone, or my hypothesis. Each page lists the exact check that would confirm or kill a lever. Added in the Nov 14 revision: the serving-first section comes from the official rules summary (llms.txt) and participants' notes. The full rules page and the serving API could not be read, and no serving score has been measured here. Everything tagged "notes" or "verify" needs checking against RULES.md before it drives work.

Sources