muse_moe_w4a16

Best public 2.420× · closing #1 2.844× · ceiling ≈3.3× (with 8 µs of fixed cost per shape) · plan target ≈3.2×

Serving fit (Nov 14 revision)

The first kernel port for the serving patch. The board is closed; what counts now is µs per served step. Stock runs Marlin here because Hopper has no MegaMoE, and Marlin alone is about half the stock layer at one request (78.8 of 171.3 µs at decode b1, 141.9 of 266.9 µs at verify b1) and more at 16 and 32.
QuestionAnswer for serving
Cheapest version firstBefore porting a kernel, try Marlin's tile and thread selection by token-count range. chenwainuo's active serving patches cap the expert tile at 16 through 256 tokens and 32 through 512. It is a few lines, easy to review, and it shows how much of the gap is configuration.
Per-M buildsCollapse the table below to two or three builds chosen by a token-count range (for example 16 and below on mma.sync, above on wgmma). One build per board shape reads as special-casing and costs patch bytes. Every M must be correct, as the tests at 4, 33 and 257 already demand.
RepackingContiguous K-range blobs, KPERM3 and 6-bit scales are lossless, but only one layout can exist: a second copy of the experts is tens of GB per GPU against a +2 GB cap. Re-layout in place after load, outside model_loader/, and measure peak memory while doing it.
NumericsE2M1 × E4M3 into BF16 is exact, so the dequant path is safe. Keep stock's rounding points and FP32 partial sums; BF16 or FP16 partials were a kernel-track allowance.
What differs from the judgeAll eight GPUs run and the step waits for the slowest, so the tail on the busiest rank is what users feel. There is no forced flush, but a layer touches far more expert bytes than L2 holds, so the bandwidth floors should still apply (to confirm).
Token counts between board sizesThe public kernels were built and timed at 1, 8, 16, 32, 128 and 256 tokens only. In serving the running batch moves, so sweep every multiple of 8 from 8 to 256 and remove the cliffs, starting with the 17–32 range where a dedicated tile ran 2.2–3.0 µs slower.
Cheap experimentsEvict-first hints on the weight stream, and one arena with fixed alignment for the re-laid-out weights. Low odds, a few lines each. See untested experiments.
Measure firstStock Marlin µs per layer at 8, 32, 128 and 256 tokens inside a served step, per GPU rank.

Plan and byte allowance: Serving track, row 2.1.

Problem

For each token, take its top-8 experts; compute only the 32 local to this GPU (EP8). Then: h = x·dq(w13[e])ᵀ → split into gate and up → y = (silu(gate)·up)·dq(w2[e])ᵀ → out[t] += g · rmsnorm(y·(1+post_norm_w)). Width H = 4096. NVFP4 weights: E2M1 codes plus one E4M3 scale per 16. BF16 activations. Output BF16 [M, 4096]. Tolerance 1.5% relative L2 vs FP32 (5× per row). Each shape has weight 60. Routing is skewed, and every test makes this GPU the busiest. Pre-timing tests include bs 4, 33, 257 and skewed 128, so builds must handle any M.

The floors fit an expert intermediate of ≈2048, about 14 MB per expert in NVFP4. That is my inference: read Config in submission.py to confirm. At M=256 all 32 local experts are touched (~450 MB, ~105 µs at 4.3 TB/s), so every scored size is weight-bandwidth bound. Compute is under half the byte time even at 256 tokens.

Roofline per shape (µs)

LoadMSGLang (Marlin)Spec floorFloor at 4.3 TB/sPlan targetTarget ×Where the time goes beyond the floor
decode 1178.811.412.7223.58~4 µs to the first 36 KB landing, A→B dependency, tail
decode 1616188.152.558.6702.69handoff dip, uneven per-expert work
decode 3232203.770.879.0902.26same, plus the first tensor-core sizes
verify 18141.937.341.6502.84ramp and tail
verify 16128459.2101.31131253.67instruction-bound dequant on hot experts (up to ~160 tokens)
verify 32256617.0108.9121.51354.57split-K atomics (up to ~77 µs in some designs), tail
Geomean3.18×vs 2.844× leader
The steady stream rate is not the problem. keell0renz's ncu sampling showed the MoE's steady DRAM rate matches a bare ring (the "MoE streams at 3.3 TB/s" story is a myth). The losses are a ~3 µs ramp, the A→B handoff dip, a slow phase B at bs1 and a 6–9 µs tail. The whole plan below is about those four, plus dequant instruction count at large M.

Design: one persistent kernel per call, specialized per M

1. In-kernel routing plan (no align or sort kernel, no grid barrier)

2. Work units and schedule

3. Weight path: TMA bulk ring, repacked, compressed scales

4. Dequant straight into wgmma A registers (swap-AB) · the main structural lever for M ≥ 17

Correction (prior art). alygg77 measured that integer ALU work does not overlap HGMMA on the same SM: 286 clocks ALU alone, 256 MMA alone, 489 together. A-from-registers saves shared-memory traffic and re-dequantization, but the real currency is instructions per weight. KPERM cut the decode loop from 20.12 to 18.44 instructions per word, and removing 12 instructions per k-block gave +5.8% on prefill experts. At d16, phase B is consumer-bound (51.9% issue utilization). Their measured MoE losses: ramp 3–6, phase-A waves 3–6, shadows 4–12, end 7–11 µs. A phase-B unit is 18.5–19.4 µs vs an end spread of 2.7–8.2 µs, so dynamic claiming can't fix the tail: split the final units finer. Weight slots saturate at 3, and NVFP4 codes are 4.000 bits (incompressible).

5. Epilogues and the A→B handoff

6. Small-M specifics (bs1, bs8)

Per-M builds

MMMATile (weights × tokens)SplitNotes
1CUDA-core FMA or mma.sync m16n864 × 8 (7 padded)N-slices across all SMsFastest first byte; tiny first stage
8mma.sync m16n8k1664 × 8stream-KRing-issuer warp variant
16, 32mma.sync or wgmma N=16/3264–128 × 16/32stream-K + DSMEM pairsBenchmark both MMAs per size
128, 256wgmma A-from-RF, 4 consumer warpgroups64 × N(tokens of the expert, ≤256)stream-K, LPT order6-bit scales, KPERM3, no atomics
other (tests)Nearest-larger build——Correctness only; untimed

Dead ends (measured by others)

Ideas beyond the public frontier (unmeasured, ranked)

  1. High. B-weight prefetch before the A→B flag (above). It hides the handoff dip, which keell0renz listed as a loss but did not remove.
  2. High. Dequant into A registers with N = all of an expert's tokens. This ends re-dequantization at v16/v32.
  3. Medium. Tail shaping: plan the stream-K so every SM's last unit is a small one. LPT plus "fill with the smallest" bin-packing cuts the 6–9 µs tail.
  4. Medium. Fused combine: fold the final weighted sum into the phase-B epilogue of the last-finishing expert per token. That removes the final pass at bs1/bs8.
  5. Low. Entropy-coded E2M1 codes. Codes are near-uniform, so the gain is small and decode is expensive.

Sources: TASK.md (spec, floors, tests); keell0renz LESSONS.md (myth, ramp and tail, 6-bit scales, stream-K, failures); alygg77 (KPERM3 gains); maximenewman (dequant op count, small-M lessons); Kushagrabainsla (atomics cost); lee101 (heaviest-first, red.release); georgxzh (SGLang µs per shape). Links on the overview.