muse_decode_layer (sets the overall rank)

Best public 2.266× · closing #1 2.614× · ceiling ≈2.9–3.1× · plan target ≈2.85×

Serving fit (Nov 14 revision)

This problem is the served step, minus communication. A served decode step is 62 of these layers at verify shapes, plus TP8/EP8 exchanges, the draft model and host time. The SGLang times in the table below imply stock layer-kernel time of about 16.6 ms per step at one request, 35.2 ms at 16 and 51.1 ms at 32 (46 sliding plus 16 full layers; cold L2 and about 80K context, so on the high side). Use those to sanity-check your stock profile.

Step model and tiered plan: Serving track. Local judge clone: Serving harness & runs.

Problem

One full layer on one GPU under TP8/EP8, with communication as identity. Returns BF16 normed output [T, 8192], the FP32 residual stream [T, 8192] and int32 routes [T, 8].

q,k,v,g = split(x @ qkv_gateᵀ)                         # 2304×8192 BF16
qk-norm, RoPE(sliding), FP8 cache write
a  = (rmsnorm(attention(q)) * sigmoid(g)) @ oᵀ          # o: 8192×1024
s  = α1·s + β1·rmsnorm(a);  h = rmsnorm(s)·(1+w)
experts = top8(sigmoid(0.177·h @ routerᵀ) + bias)       # FP32 256×8192
y  = local_experts(rmsnorm(h @ poolerᵀ)·(1+w))          # pooler 4096×8192, NVFP4 experts
f  = y @ unpoolerᵀ                                     # only unpooler[:1024] matters at T ≤ 8
s  = α2·s + β2·rmsnorm(f·(1+w));  out = rmsnorm(s)·(1+w)

The router check is discrete. Picks must be within 0.05 of the reference's 8th-best score; clear winners must be present; route and distinct-expert counts must be within limits. The MoE output is then checked with your picks.

Shapes and targets (µs)

LoadSGLang sliding / fullFloor sliding / fullTarget sliding / fullTarget ×
decode b1171.3 / 181.940.0 / 44.260 / 662.83
decode b16259.3 / 442.576.7 / 137.5105 / 1702.52
decode b32271.0 / 623.884.5 / 208.1115 / 2452.44
verify b1266.9 / 270.462.3 / 66.485 / 923.09
verify b16523.5 / 694.4126.6 / 188.7175 / 2452.94
verify b32732.0 / 1,086.8143.8 / 267.2215 / 3403.33
Geomean (load = 46·sliding + 16·full)2.84×

Byte budget at decode b1 sliding (why ~60 µs is possible)

OpBytes (12-bit where BF16)@4.3 TB/sBound
qkv_gate GEMV28.3 MB6.6 µsbandwidth
attention, sliding0.5 MB—latency (~4 µs)
o GEMV + residual/norm12.6 MB2.9 µsbandwidth + cross-CTA norm
router (FP32) + pooler8.4 + 50.3 MB13.7 µsbandwidth
top-8 + MoE (≈4 local experts)≈57 MB≈13 µsbandwidth + ramp/tail
unpooler ([:1024]) + residual 2 + norm≈6.3 MB1.5 µslatency
Sum≈163 MB≈38 µs+ ~8 dependency boundaries × 1–3 µs

The bandwidth part is ~38 µs; the rest is boundaries. Each kernel edge costs a drain, a ramp to the first landed byte (~4 µs cold) and a cross-SM dependency. Winning the layer means overlapping weight streaming with those boundaries.

Architecture: a PDL chain of ~8 kernels, built from the problem winners

Start from keell0renz's layer: about 9 kernels with PDL, sk2 12-bit GEMVs, pooler and router sharing one launch, Triton routing, the persistent MoE with per-batch builds and stream-K at ≥17 tokens, and a cluster-based residual/norm boundary. Then swap in the new attention and MoE from the problem pages, and apply the levers below.

Correction (second research pass, prior art). alygg77 measured most of Levers A and B and they lost. An early pooler/router stream at d32s was 2.1–7.7 µs slower. L2 prefetch from sliding attention was 2.6–5.1 µs slower. Side-stream junctions cost ≥2 µs each. Two pre-wait weight stages is the optimum, and 4–6 slow attention by 1.7 µs. Their budgets also show where the time actually is: d32s 154 µs = MoE 84 + dense GEMV 51.5 (floor 34.9) + glue 8.8 + attention 9.8; v32s 306 µs = MoE 188 + dense 83 + attention 14.8 + glue 18.4, with <1.1 µs idle between kernels. The decode-layer plan is therefore: port the best MoE (by far the largest term), then close the dense-GEMV gap to its floor, then attention. Treat A and B below as low-EV experiments: only the in-launch router-first variant is untested.

Lever A: weights don't depend on activations, so stream them early, but only behind latency-bound ops

Lever B: router first, pooler concurrent (reorder, my hypothesis)

The router needs only h. The pooler output feeds the experts, but the expert identities come from the router alone. Today the router shares a launch with the pooler, and routing runs after both. Instead:

  1. One launch: a few CTAs compute the exact FP32 router (8.4 MB, ~2–3 µs) first and run top-8 in-kernel (sigmoid + bias + top-8 over 256 is one warp per token) while the remaining CTAs stream the pooler.
  2. The routes and the per-expert plan are published via a flag well before the pooler finishes. MoE CTAs launched with PDL start streaming expert weights (phase A) during the pooler's tail and its rmsnorm boundary, then wait on the pooler-output flag.

This takes the routing kernel and the MoE ramp (~3–4 µs cold to first byte) off the critical path. Expected 3–6 µs per shape. Risk: bandwidth contention with the pooler stream. Pace MoE prefetch to the pooler's tail only (last ~20% of its units).

Lever C: fold norms and boundaries into neighbors

Lever D: cold-code and cold-constant warming across the chain

Why not a single megakernel here?

keell0renz measured a persistent whole-layer kernel below chain + PDL, because cross-SM hops get 3–6× slower under HBM saturation. LMSYS's megakernel wins at batch 1 on 8×GB300 by eliminating ~1,450 launches per step and fusing communication. On this single-GPU judge with comm = identity, the chain is the better bet. A three-segment compromise (attention block · dense block · MoE+unpooler block) is a medium-EV experiment if the boundaries above still show >2 µs gaps on the per-SM timeline.

Batch-specific notes

Validation

Sources: TASK.md (spec, floors, router check); georgxzh (per-shape SGLang µs); keell0renz (chain design, PDL trigger, megakernel failure); Kushagrabainsla (norm folding, L2-prefetch slowdown); infraax (prefetch congestion); trxphxle (pre-warm); LMSYS megakernel (icache, prefetch, sync costs); yesthatwilly (ZL guard). Links on the overview.