Methodology: clone the judge, then measure everything
Most of the late-stage work by top teams found kernels already at a hardware limit they hadn't measured. Measure the ceiling first, and decide only on paired runs that copy the judge.
Scope (Nov 14 revision). This page describes the kernel judge, which closed on Nov 5. Serving is judged differently: a patch on SGLang v0.5.23, a whole 8×H200 node, throughput at stock's per-user speed over four loads, and at most seven scored runs left. The local clone for that is on Serving harness & runs.
- Does not carry over: the 1 GiB flush before every call, the 4 µs subtraction, equal weight for tiny shapes, variant sweeps in
prepare(), the Nov 3 precision allowances (serving allows nothing below stock), and thedev-muse-spark-openimage (serving useslmsysorg/sglang:v0.5.23). - Carries over: paired A/B runs on the same hardware, "predict, then kill", a same-codegen control, per-SM timelines, the bare-ring floor, and the measured H200 limits below as upper bounds.
What the judge does
| Mechanic | Detail | Consequence |
|---|---|---|
| Interface | One Python file. prepare(weights)→state runs untimed (300 s limit); kernel_fn(data, state) is timed. The first line must be # okc: problem=…. Image: lmsysorg/sglang:dev-muse-spark-open. | All repacking, encoding and variant compilation goes in prepare. The whole run, compilation included, has a 15-minute cap. |
| Timing | Each shape is captured as a CUDA graph and replayed, alternating with SGLang's replays on the same fresh inputs and the same GPU. Order swaps between reps. Per shape: median of paired ratios. | No host syncs, no allocation, no branching on values. Shape branching is fine. |
| Cold L2 | Before each replay the judge reads ~1 GiB (~260 µs). Weights and code never stay cached between calls. | Every first touch is a DRAM trip: prologues, page tables, scales, instruction fetch. |
| Fixed-cost subtraction | ~4 µs (≈4.1 measured) is removed from both sides. | A 9 µs shape counts as 5 µs, so launch tails weigh heavily. |
| Score | Load time = Σ weight × shape time. Problem score = geomean over loads of SGLang÷yours, with each load equal. The overall board ranks by the decode-step speedup your files would give together. | The tiny batch-1 loads count as much as batch 32. |
| Correctness | Relative L2 vs FP32 reference, per-row bound 5× (near-zero rows measured against 0.1× the mean row). Checked on pre-timing tests and every timed call. | No answer caches: fresh inputs, all outputs checked. |
| Precision rules | Weights lossless only (NVFP4 stays NVFP4; BF16 stays BF16 or smaller losslessly). BF16 activations where SGLang uses them. FP8 KV. Queries may round to FP8 E4M3. Since Nov 3: approximate math accurate to BF16, BF16/FP16 partial sums (split-K, split-KV), and FP16 accumulation of QKᵀ and P·V are allowed. Reported, not confirmed: P stored as FP8 at a power-of-two scale with approximated exp2. | Opens FP8 QKᵀ, FP16-accumulated PV and cheaper split partials. |
| Source rules | Since Nov 2: readable source only, with no embedded binaries or compressed or obfuscated code. CUDA as strings compiled with load_inline/NVRTC. | The 12-bit weight encoding is fine; compressed source payloads are not. |
Local judge clone
A minimal skeleton that has tracked the real judge within a few % for several teams. Put both sides in their own graphs and pair every rep.
import torch, statistics as st
FLUSH = torch.empty(1 << 30, dtype=torch.uint8, device="cuda") # > 50 MB L2; ~260 us to read
def flush(): FLUSH.sum(dtype=torch.int32) # a read, not a write
def capture(fn, *args):
s = torch.cuda.Stream(); s.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(s):
for _ in range(3): fn(*args) # warm and JIT outside capture
torch.cuda.current_stream().wait_stream(s)
g = torch.cuda.CUDAGraph()
with torch.cuda.graph(g): out = fn(*args)
return g, out
def time_graph(g, refill):
refill() # fresh inputs written in place (graph has static ptrs)
flush()
a, b = torch.cuda.Event(True), torch.cuda.Event(True)
a.record(); g.replay(); b.record(); b.synchronize()
return a.elapsed_time(b) * 1e3 - 4.1 # us, minus fixed replay cost
def paired_speedup(g_sgl, g_you, refill, reps=60):
r = []
for i in range(reps):
if i % 2 == 0: ts, ty = time_graph(g_sgl, refill), time_graph(g_you, refill)
else: ty, ts = time_graph(g_you, refill), time_graph(g_sgl, refill)
r.append(ts / ty)
return st.median(r)
# per load: speedup = sum(w * t_sgl) / sum(w * t_you), then geomean over loads
# also run reference checks on EVERY timed call's output, not just warmup
- Fresh inputs: regenerate with new seeds per rep. Routing changes every call; skewed popularity is part of the generator.
- Same GPU class only. H100 results didn't carry over for MoE (3.35 vs 4.8 TB/s, different L2). Rent an H200 SXM with the judge image.
- Use the judge's checks: copy
reference.py/checks.pyfromokc initinto the harness. Include the 16 required attention tests (contexts up to 1,048,576, page sizes 1 and 64, KV scale 1/64).
Measured H200 facts (use these, not the spec sheet)
| Quantity | Value | Source / note |
|---|---|---|
| HBM read ceiling, TMA bulk ring | 4.2–4.45 TB/s | Spec 4.8. Reached at 132 CTAs × 4 slots × 36 KB in flight. More depth didn't help. |
| DRAM ceiling vs read size (alygg77) | 4.60 @ 2 GiB · 4.33 @ 256 MiB · 3.6–3.8 @ 66 MiB | Use a size-matched ceiling for each shape's floor. |
| Per-SM write to L2 · DSMEM read | 40–50 GB/s · ~30 GB/s | Partial-sum writes are expensive. Prefer in-register or in-CTA reductions. |
| Integer ALU ∥ HGMMA | 286 + 256 → 489 clk | Dequant and decode don't overlap wgmma on the same SM. |
| Empty graph replay | 4.70–4.74 µs | Slightly more than the judge's ~4 µs subtraction. |
| Memory-layout sensitivity | 1.5–2.5 µs at v32 | The prepare() allocation pad moves results. Test ≥4 pads before trusting a <1.5 µs gain. |
| prepare() time | 273–312 s seen | The limit is 300 s. Big variant sweeps risk timeouts. |
| First 36 KB unit lands after launch | ~4 µs | Then one 36 KB unit per SM every ~1.09 µs. Small reads can't reach peak. |
| Cross-SM flag hop (132 CTAs) | 1.4 µs idle / 6.3 µs under load | Sync is 3–6× slower under HBM saturation. |
| GPU-scope acquire poll | 141–145 ns | LMSYS (GB300). Polls queue behind bulk copies for 1.0–2.6 µs. |
| GPU-wide fence | 0.4 µs idle / 3.6–4.0 µs with TMA in flight | Use sentinel-based, fence-free handoffs where possible. |
| SM clock under dequant + MMA | ~1.81 GHz | 1.97 GHz on a bare ring; the stream rate didn't change. |
| Instruction cache per SM | ~128 KB | Two 100 KB straight-line blocks: 19.5 µs together vs 12.4 µs each. |
| MUFU ex2 throughput | 16/clk/SM | ≈3.8 T exp/s at 1.8 GHz. Binding only for full verify at batch 32 (~328M exps ≈ 86 µs vs ~152 µs of KV bytes). |
| Little's law for full BW | ≈33 KB in flight per SM | 4.3 TB/s × ~1 µs ÷ 132 SMs. Gathers (page size 1) need ≥ that many 128 B row requests outstanding. |
Noise budget and decision rules
- Run-to-run noise: 1σ ≈ 0.63% per board run; two judge clusters differ by 0.4–1%. Seeing a real 1% change takes 6–8 runs per arm. For decisions under 1%, use 30+ paired reps locally.
- "Predict, then kill": write the predicted number and a kill threshold before each run (infraax). Promote only if the result beats the incumbent outside its confidence interval.
- Compare against a same-codegen control. nvcc can shift a hot loop by ~5% from unrelated edits. Check SASS diffs (
cuobjdump -sass) for spills and sunk loads.
Experiment throughput tricks
- Variant sweeps inside
prepare(). Compile 10–40 variants and time them back to back on the judge's GPU, so one run becomes one experiment table. Mind the 15-minute cap: compile concurrently (threads plus NVRTC) or warm a shared JIT cache. - JIT on CPU-only containers into a shared cache, so GPU hours go only to timing.
- Per-SM timelines. Stamp
%globaltimerand%smidat phase boundaries per CTA (lane 0, plain stores) and plot start, wait and end per SM. Every big win came from finding the one slow phase.ncuwas unavailable on rented pods. - Bare-ring skeleton. Before tuning a kernel, run a kernel that only streams the same bytes with the same CTA count. That is your real floor.
CUDA-graph and runtime gotchas
torch.zeros/torch.emptyinside the timed function becomes a graph node or allocation. Preallocate inprepareand keep self-resetting counters (the last CTA resets them).torch.cuda.Stream()can hand back a pooled stream that is already capturing. Captured memory must stay owned by a live graph or pool.- Fresh OS threads: bind the CUDA context per thread (
cuCtxSetCurrent) before creating TMA descriptors. This fixed TMA descriptor failures and the prefill "turn 1 full" mis-route. - Implement
__deepcopy__on state objects so each copy owns its weights (speed-neutral, robustness). - Kernels launched with PDL before their predecessor finishes must read predecessor output with
ld.global.cg.
Correctness traps
- Restore SGLang's BF16 rounding points (QK-norm, RoPE, cache scaling, attention output, expert GEMM outputs, pooler input and output). Leaving them out drifts toward the per-row bound (trxphxle).
- The router check is discrete. Each picked expert must be within 0.05 of the reference's 8th-best score, and clear winners must be picked. Keep router math exact FP32 at T≤8 (alygg77's reading of the rules); hi/lo BF16 pairs are allowed above that.
- Attention tests put keys at window edges, give keys within a head a shared direction (large-magnitude logits) and use page size 64 with KV scale 1/64. Handle page size and scale generically even though scored shapes use 1 and 1.
Sources: keell0renz LESSONS.md; Kushagrabainsla; infraax; alygg77; maximenewman; LMSYS megakernel post; georgxzh rule timeline; TASK.md specs (links on the overview).