muse_decode_layer (sets the overall rank)
Best public 2.266× · closing #1 2.614× · ceiling ≈2.9–3.1× · plan target ≈2.85×
Serving fit (Nov 14 revision)
- Port by module, not as a chain. The correction box below already gives the order: experts, then dense GEMVs, then attention. In SGLang each is its own interface (fused MoE, linear layers, attention backend), so each can be a reviewable diff with its own A/B. A whole-layer file is neither.
- Communication is not identity. Kernel-to-kernel PDL edges now cross all-reduce and all-gather points, and the step waits for the slowest GPU. Re-measure every boundary claim on this page inside a served step.
- Skip Levers A and B in serving. They were measured as losses on the judge, and they add code. Lever C (norm folding) is usable only if stock's BF16 rounding points are preserved. Lever D targets the judge's flush; re-measure before spending bytes on it.
- The router check becomes the agreement gate. One different expert pick changes the output. Keep the router in exact FP32 and fail closed.
- Fixed cost is what the score amplifies. Launch count, gaps between kernels and weight bytes read once per layer do not grow with users. Under the plain reading of the score they count several times over at loads 1 and 4 (sketch).
- Check the toolchain before porting anything. The public kernels compile native CUDA at start-up; confirm that path and its compile time in the v0.5.23 image (untested experiments, row 6).
- Robustness is worth more here than on the boards. The per-thread context bind, deepcopy-safe holders and the no-local-route guard prevent hangs, and a hung run costs one of about seven scored runs.
Step model and tiered plan: Serving track. Local judge clone: Serving harness & runs.
Problem
One full layer on one GPU under TP8/EP8, with communication as identity. Returns BF16 normed output [T, 8192], the FP32 residual stream [T, 8192] and int32 routes [T, 8].
q,k,v,g = split(x @ qkv_gateᵀ) # 2304×8192 BF16
qk-norm, RoPE(sliding), FP8 cache write
a = (rmsnorm(attention(q)) * sigmoid(g)) @ oᵀ # o: 8192×1024
s = α1·s + β1·rmsnorm(a); h = rmsnorm(s)·(1+w)
experts = top8(sigmoid(0.177·h @ routerᵀ) + bias) # FP32 256×8192
y = local_experts(rmsnorm(h @ poolerᵀ)·(1+w)) # pooler 4096×8192, NVFP4 experts
f = y @ unpoolerᵀ # only unpooler[:1024] matters at T ≤ 8
s = α2·s + β2·rmsnorm(f·(1+w)); out = rmsnorm(s)·(1+w)
The router check is discrete. Picks must be within 0.05 of the reference's 8th-best score; clear winners must be present; route and distinct-expert counts must be within limits. The MoE output is then checked with your picks.
Shapes and targets (µs)
| Load | SGLang sliding / full | Floor sliding / full | Target sliding / full | Target × |
|---|---|---|---|---|
| decode b1 | 171.3 / 181.9 | 40.0 / 44.2 | 60 / 66 | 2.83 |
| decode b16 | 259.3 / 442.5 | 76.7 / 137.5 | 105 / 170 | 2.52 |
| decode b32 | 271.0 / 623.8 | 84.5 / 208.1 | 115 / 245 | 2.44 |
| verify b1 | 266.9 / 270.4 | 62.3 / 66.4 | 85 / 92 | 3.09 |
| verify b16 | 523.5 / 694.4 | 126.6 / 188.7 | 175 / 245 | 2.94 |
| verify b32 | 732.0 / 1,086.8 | 143.8 / 267.2 | 215 / 340 | 3.33 |
| Geomean (load = 46·sliding + 16·full) | 2.84× | |||
Byte budget at decode b1 sliding (why ~60 µs is possible)
| Op | Bytes (12-bit where BF16) | @4.3 TB/s | Bound |
|---|---|---|---|
| qkv_gate GEMV | 28.3 MB | 6.6 µs | bandwidth |
| attention, sliding | 0.5 MB | — | latency (~4 µs) |
| o GEMV + residual/norm | 12.6 MB | 2.9 µs | bandwidth + cross-CTA norm |
| router (FP32) + pooler | 8.4 + 50.3 MB | 13.7 µs | bandwidth |
| top-8 + MoE (≈4 local experts) | ≈57 MB | ≈13 µs | bandwidth + ramp/tail |
| unpooler ([:1024]) + residual 2 + norm | ≈6.3 MB | 1.5 µs | latency |
| Sum | ≈163 MB | ≈38 µs | + ~8 dependency boundaries × 1–3 µs |
The bandwidth part is ~38 µs; the rest is boundaries. Each kernel edge costs a drain, a ramp to the first landed byte (~4 µs cold) and a cross-SM dependency. Winning the layer means overlapping weight streaming with those boundaries.
Architecture: a PDL chain of ~8 kernels, built from the problem winners
Start from keell0renz's layer: about 9 kernels with PDL, sk2 12-bit GEMVs, pooler and router sharing one launch, Triton routing, the persistent MoE with per-batch builds and stream-K at ≥17 tokens, and a cluster-based residual/norm boundary. Then swap in the new attention and MoE from the problem pages, and apply the levers below.
Lever A: weights don't depend on activations, so stream them early, but only behind latency-bound ops
- With PDL, the dependent kernel's CTAs start as soon as SMs free up. Each producer warp issues its first ring stages of weights before
griddepcontrol.wait, and the consumers wait. - Measured caveat: a blanket "tools before wait" prefetch measured +2 µs on the light path and −3.9 µs on the full path from congestion (infraax). L2 prefetch of next-layer weights during attention made attention slower (Kushagrabainsla). So apply it only where the predecessor is latency-bound and uses few SMs:
- o-proj during sliding attention. Attention at b1 uses ≤32 CTAs for ~4 µs; o's 12.6 MB (12-bit) fits in shared memory across ~100 idle SMs (≈22 MB of shared memory). Prefetch into shared memory, not L2, so the attention CTAs' gathers don't queue behind it. Pace it. Expected 2–3 µs per sliding shape.
- MoE weights during top-8 selection. See Lever B.
- Unpooler during the MoE tail (6–9 µs, latency-ish): 6 MB into shared memory on SMs that already finished their MoE units.
- Trigger
launch_dependentsright after the producer's last weight issue, not at kernel end: +0.9% per layer.
Lever B: router first, pooler concurrent (reorder, my hypothesis)
The router needs only h. The pooler output feeds the experts, but the expert identities come from the router alone. Today the router shares a launch with the pooler, and routing runs after both. Instead:
- One launch: a few CTAs compute the exact FP32 router (8.4 MB, ~2–3 µs) first and run top-8 in-kernel (sigmoid + bias + top-8 over 256 is one warp per token) while the remaining CTAs stream the pooler.
- The routes and the per-expert plan are published via a flag well before the pooler finishes. MoE CTAs launched with PDL start streaming expert weights (phase A) during the pooler's tail and its rmsnorm boundary, then wait on the pooler-output flag.
This takes the routing kernel and the MoE ramp (~3–4 µs cold to first byte) off the critical path. Expected 3–6 µs per shape. Risk: bandwidth contention with the pooler stream. Pace MoE prefetch to the pooler's tail only (last ~20% of its units).
Lever C: fold norms and boundaries into neighbors
- Pre-expert RMSNorm: the pooler's epilogue emits per-token sum of squares; MoE gemm1 applies the scale while staging x (Kushagrabainsla, T ≤ 32).
- Residual 1 → h: a cluster boundary kernel (keell0renz). Or fuse into the o-proj's last-CTA epilogue for T ≤ 8, where one CTA can own whole rows after a DSMEM reduction.
- Pad/remap glue kernels removed entirely. Glue fusions beyond that measured ≈0 for infraax, so stop once the launches are gone.
Lever D: cold-code and cold-constant warming across the chain
- The judge flushes L2, and with it instructions. The LMSYS megakernel pulls its own code into L2 each launch (one word per 128 B line,
evict_last) and does a dry run of latency-critical paths: 0.5–1 µs per op. Here, the cheapest version is small code (rolled loops, shared device functions across per-M builds) and an early kernel prefetching the constant tables (norm weights, RoPE, scale LUTs) that later kernels touch first. "Pre-warmed code and constants for every D1/V1 kernel" measured +0.16% for trxphxle, so the code side is small; constants on the critical path are the part worth doing.
Why not a single megakernel here?
keell0renz measured a persistent whole-layer kernel below chain + PDL, because cross-SM hops get 3–6× slower under HBM saturation. LMSYS's megakernel wins at batch 1 on 8×GB300 by eliminating ~1,450 launches per step and fusing communication. On this single-GPU judge with comm = identity, the chain is the better bet. A three-segment compromise (attention block · dense block · MoE+unpooler block) is a medium-EV experiment if the boundaries above still show >2 µs gaps on the per-SM timeline.
Batch-specific notes
- T ≤ 8: unpooler reads only
unpooler[:1024]. Router is exact FP32. MoE is the bs1/bs8 builds. - T = 128/256 (verify b16/b32): projections switch to wgmma (pinned plans or custom). The attention full-verify kernel decides the full shapes. Routing is skewed, so the MoE LPT order matters most here.
- Full vs sliding: only attention differs, so share everything else and keep one chain with an attention-kernel switch.
Validation
- Router check on 1,000 fresh seeds offline (margins 0.05 / 0.03 / 0.04, counts). Fail closed: if a fast path ever disagrees, use exact FP32.
- Per-SM timeline of the whole chain at d1 sliding. Target: no SM idle more than ~1.5 µs between its last unit of one kernel and its first landed byte of the next.
- Robustness: per-thread CUDA-context bind, deepcopy-safe cuBLASLt holders and the ZL guard for launches with no local route (yesthatwilly's hang fix).
Sources: TASK.md (spec, floors, router check); georgxzh (per-shape SGLang µs); keell0renz (chain design, PDL trigger, megakernel failure); Kushagrabainsla (norm folding, L2-prefetch slowdown); infraax (prefetch congestion); trxphxle (pre-warm); LMSYS megakernel (icache, prefetch, sync costs); yesthatwilly (ZL guard). Links on the overview.