Methodology: clone the judge, then measure everything

Most of the late-stage work by top teams found kernels already at a hardware limit they hadn't measured. Measure the ceiling first, and decide only on paired runs that copy the judge.

Scope (Nov 14 revision). This page describes the kernel judge, which closed on Nov 5. Serving is judged differently: a patch on SGLang v0.5.23, a whole 8×H200 node, throughput at stock's per-user speed over four loads, and at most seven scored runs left. The local clone for that is on Serving harness & runs.

What the judge does

MechanicDetailConsequence
InterfaceOne Python file. prepare(weights)→state runs untimed (300 s limit); kernel_fn(data, state) is timed. The first line must be # okc: problem=…. Image: lmsysorg/sglang:dev-muse-spark-open.All repacking, encoding and variant compilation goes in prepare. The whole run, compilation included, has a 15-minute cap.
TimingEach shape is captured as a CUDA graph and replayed, alternating with SGLang's replays on the same fresh inputs and the same GPU. Order swaps between reps. Per shape: median of paired ratios.No host syncs, no allocation, no branching on values. Shape branching is fine.
Cold L2Before each replay the judge reads ~1 GiB (~260 µs). Weights and code never stay cached between calls.Every first touch is a DRAM trip: prologues, page tables, scales, instruction fetch.
Fixed-cost subtraction~4 µs (≈4.1 measured) is removed from both sides.A 9 µs shape counts as 5 µs, so launch tails weigh heavily.
ScoreLoad time = Σ weight × shape time. Problem score = geomean over loads of SGLang÷yours, with each load equal. The overall board ranks by the decode-step speedup your files would give together.The tiny batch-1 loads count as much as batch 32.
CorrectnessRelative L2 vs FP32 reference, per-row bound 5× (near-zero rows measured against 0.1× the mean row). Checked on pre-timing tests and every timed call.No answer caches: fresh inputs, all outputs checked.
Precision rulesWeights lossless only (NVFP4 stays NVFP4; BF16 stays BF16 or smaller losslessly). BF16 activations where SGLang uses them. FP8 KV. Queries may round to FP8 E4M3. Since Nov 3: approximate math accurate to BF16, BF16/FP16 partial sums (split-K, split-KV), and FP16 accumulation of QKᵀ and P·V are allowed. Reported, not confirmed: P stored as FP8 at a power-of-two scale with approximated exp2.Opens FP8 QKᵀ, FP16-accumulated PV and cheaper split partials.
Source rulesSince Nov 2: readable source only, with no embedded binaries or compressed or obfuscated code. CUDA as strings compiled with load_inline/NVRTC.The 12-bit weight encoding is fine; compressed source payloads are not.

Local judge clone

A minimal skeleton that has tracked the real judge within a few % for several teams. Put both sides in their own graphs and pair every rep.

import torch, statistics as st
FLUSH = torch.empty(1 << 30, dtype=torch.uint8, device="cuda")   # > 50 MB L2; ~260 us to read
def flush(): FLUSH.sum(dtype=torch.int32)                         # a read, not a write

def capture(fn, *args):
    s = torch.cuda.Stream(); s.wait_stream(torch.cuda.current_stream())
    with torch.cuda.stream(s):
        for _ in range(3): fn(*args)                               # warm and JIT outside capture
    torch.cuda.current_stream().wait_stream(s)
    g = torch.cuda.CUDAGraph()
    with torch.cuda.graph(g): out = fn(*args)
    return g, out

def time_graph(g, refill):
    refill()                                     # fresh inputs written in place (graph has static ptrs)
    flush()
    a, b = torch.cuda.Event(True), torch.cuda.Event(True)
    a.record(); g.replay(); b.record(); b.synchronize()
    return a.elapsed_time(b) * 1e3 - 4.1          # us, minus fixed replay cost

def paired_speedup(g_sgl, g_you, refill, reps=60):
    r = []
    for i in range(reps):
        if i % 2 == 0: ts, ty = time_graph(g_sgl, refill), time_graph(g_you, refill)
        else:          ty, ts = time_graph(g_you, refill), time_graph(g_sgl, refill)
        r.append(ts / ty)
    return st.median(r)

# per load: speedup = sum(w * t_sgl) / sum(w * t_you), then geomean over loads
# also run reference checks on EVERY timed call's output, not just warmup

Measured H200 facts (use these, not the spec sheet)

QuantityValueSource / note
HBM read ceiling, TMA bulk ring4.2–4.45 TB/sSpec 4.8. Reached at 132 CTAs × 4 slots × 36 KB in flight. More depth didn't help.
DRAM ceiling vs read size (alygg77)4.60 @ 2 GiB · 4.33 @ 256 MiB · 3.6–3.8 @ 66 MiBUse a size-matched ceiling for each shape's floor.
Per-SM write to L2 · DSMEM read40–50 GB/s · ~30 GB/sPartial-sum writes are expensive. Prefer in-register or in-CTA reductions.
Integer ALU ∥ HGMMA286 + 256 → 489 clkDequant and decode don't overlap wgmma on the same SM.
Empty graph replay4.70–4.74 µsSlightly more than the judge's ~4 µs subtraction.
Memory-layout sensitivity1.5–2.5 µs at v32The prepare() allocation pad moves results. Test ≥4 pads before trusting a <1.5 µs gain.
prepare() time273–312 s seenThe limit is 300 s. Big variant sweeps risk timeouts.
First 36 KB unit lands after launch~4 µsThen one 36 KB unit per SM every ~1.09 µs. Small reads can't reach peak.
Cross-SM flag hop (132 CTAs)1.4 µs idle / 6.3 µs under loadSync is 3–6× slower under HBM saturation.
GPU-scope acquire poll141–145 nsLMSYS (GB300). Polls queue behind bulk copies for 1.0–2.6 µs.
GPU-wide fence0.4 µs idle / 3.6–4.0 µs with TMA in flightUse sentinel-based, fence-free handoffs where possible.
SM clock under dequant + MMA~1.81 GHz1.97 GHz on a bare ring; the stream rate didn't change.
Instruction cache per SM~128 KBTwo 100 KB straight-line blocks: 19.5 µs together vs 12.4 µs each.
MUFU ex2 throughput16/clk/SM≈3.8 T exp/s at 1.8 GHz. Binding only for full verify at batch 32 (~328M exps ≈ 86 µs vs ~152 µs of KV bytes).
Little's law for full BW≈33 KB in flight per SM4.3 TB/s × ~1 µs ÷ 132 SMs. Gathers (page size 1) need ≥ that many 128 B row requests outstanding.

Noise budget and decision rules

Experiment throughput tricks

  1. Variant sweeps inside prepare(). Compile 10–40 variants and time them back to back on the judge's GPU, so one run becomes one experiment table. Mind the 15-minute cap: compile concurrently (threads plus NVRTC) or warm a shared JIT cache.
  2. JIT on CPU-only containers into a shared cache, so GPU hours go only to timing.
  3. Per-SM timelines. Stamp %globaltimer and %smid at phase boundaries per CTA (lane 0, plain stores) and plot start, wait and end per SM. Every big win came from finding the one slow phase. ncu was unavailable on rented pods.
  4. Bare-ring skeleton. Before tuning a kernel, run a kernel that only streams the same bytes with the same CTA count. That is your real floor.

CUDA-graph and runtime gotchas

Correctness traps

Sources: keell0renz LESSONS.md; Kushagrabainsla; infraax; alygg77; maximenewman; LMSYS megakernel post; georgxzh rule timeline; TASK.md specs (links on the overview).