muse_moe_w4a16
Best public 2.420× · closing #1 2.844× · ceiling ≈3.3× (with 8 µs of fixed cost per shape) · plan target ≈3.2×
Serving fit (Nov 14 revision)
| Question | Answer for serving |
|---|---|
| Cheapest version first | Before porting a kernel, try Marlin's tile and thread selection by token-count range. chenwainuo's active serving patches cap the expert tile at 16 through 256 tokens and 32 through 512. It is a few lines, easy to review, and it shows how much of the gap is configuration. |
| Per-M builds | Collapse the table below to two or three builds chosen by a token-count range (for example 16 and below on mma.sync, above on wgmma). One build per board shape reads as special-casing and costs patch bytes. Every M must be correct, as the tests at 4, 33 and 257 already demand. |
| Repacking | Contiguous K-range blobs, KPERM3 and 6-bit scales are lossless, but only one layout can exist: a second copy of the experts is tens of GB per GPU against a +2 GB cap. Re-layout in place after load, outside model_loader/, and measure peak memory while doing it. |
| Numerics | E2M1 × E4M3 into BF16 is exact, so the dequant path is safe. Keep stock's rounding points and FP32 partial sums; BF16 or FP16 partials were a kernel-track allowance. |
| What differs from the judge | All eight GPUs run and the step waits for the slowest, so the tail on the busiest rank is what users feel. There is no forced flush, but a layer touches far more expert bytes than L2 holds, so the bandwidth floors should still apply (to confirm). |
| Token counts between board sizes | The public kernels were built and timed at 1, 8, 16, 32, 128 and 256 tokens only. In serving the running batch moves, so sweep every multiple of 8 from 8 to 256 and remove the cliffs, starting with the 17–32 range where a dedicated tile ran 2.2–3.0 µs slower. |
| Cheap experiments | Evict-first hints on the weight stream, and one arena with fixed alignment for the re-laid-out weights. Low odds, a few lines each. See untested experiments. |
| Measure first | Stock Marlin µs per layer at 8, 32, 128 and 256 tokens inside a served step, per GPU rank. |
Plan and byte allowance: Serving track, row 2.1.
Problem
For each token, take its top-8 experts; compute only the 32 local to this GPU (EP8). Then: h = x·dq(w13[e])ᵀ → split into gate and up → y = (silu(gate)·up)·dq(w2[e])ᵀ → out[t] += g · rmsnorm(y·(1+post_norm_w)). Width H = 4096. NVFP4 weights: E2M1 codes plus one E4M3 scale per 16. BF16 activations. Output BF16 [M, 4096]. Tolerance 1.5% relative L2 vs FP32 (5× per row). Each shape has weight 60. Routing is skewed, and every test makes this GPU the busiest. Pre-timing tests include bs 4, 33, 257 and skewed 128, so builds must handle any M.
The floors fit an expert intermediate of ≈2048, about 14 MB per expert in NVFP4. That is my inference: read Config in submission.py to confirm. At M=256 all 32 local experts are touched (~450 MB, ~105 µs at 4.3 TB/s), so every scored size is weight-bandwidth bound. Compute is under half the byte time even at 256 tokens.
Roofline per shape (µs)
| Load | M | SGLang (Marlin) | Spec floor | Floor at 4.3 TB/s | Plan target | Target × | Where the time goes beyond the floor |
|---|---|---|---|---|---|---|---|
| decode 1 | 1 | 78.8 | 11.4 | 12.7 | 22 | 3.58 | ~4 µs to the first 36 KB landing, A→B dependency, tail |
| decode 16 | 16 | 188.1 | 52.5 | 58.6 | 70 | 2.69 | handoff dip, uneven per-expert work |
| decode 32 | 32 | 203.7 | 70.8 | 79.0 | 90 | 2.26 | same, plus the first tensor-core sizes |
| verify 1 | 8 | 141.9 | 37.3 | 41.6 | 50 | 2.84 | ramp and tail |
| verify 16 | 128 | 459.2 | 101.3 | 113 | 125 | 3.67 | instruction-bound dequant on hot experts (up to ~160 tokens) |
| verify 32 | 256 | 617.0 | 108.9 | 121.5 | 135 | 4.57 | split-K atomics (up to ~77 µs in some designs), tail |
| Geomean | 3.18× | vs 2.844× leader | |||||
Design: one persistent kernel per call, specialized per M
1. In-kernel routing plan (no align or sort kernel, no grid barrier)
- Every CTA reads
topk_idsand weights (≤ 2,048 entries, a single coalesced read) and redundantly builds the same (expert → token list) table in shared memory with warp-aggregated atomics. It's tiny, and it saves a launch plus a cross-SM sync (1.4–6.3 µs). - Each CTA then computes the same static stream-K schedule from a per-unit cost table, using a warp-parallel shuffle-scan of job costs. A serial prologue was the critical path; parallelizing it gave +0.9% at 32 tokens and +0.5% at 128.
2. Work units and schedule
- Phase A unit = (expert, 2I-column tile, K-range); phase B unit = (expert, 4096-column tile, K-range). Cost = weight bytes + α·tokens·tile. Calibrate α on the judge-like setup: a 12.5% cost-table error cost 7–9%.
- Order units heaviest-first (LPT) so hot experts don't form the tail (lee101 v14: heaviest-first tile queue). Static stream-K beat dynamic atomic queues under load.
- Split-K only where the expert count is too small to fill 132 SMs (bs1/bs8). Reduce partners over DSMEM in a 2-CTA cluster or a single owner fix-up, never per-element
red.global.add(that cost up to ~77 µs at 256 tokens for Kushagrabainsla; vector atomics didn't help).
3. Weight path: TMA bulk ring, repacked, compressed scales
- In
prepare(), repack so each unit's K-range is one contiguous blob. A singlecp.async.bulkper stage, ring of 4 × ~36 KB, one producer warp with mbarriers. - KPERM3-style K permutation so one scale register pair covers four 16-wide k-steps. That stopped register spills: +2.6% at d32, +3.9% at v16, +3.1% at v32 (alygg77).
- Lossless 6-bit scale indices with a per-tensor 64-entry E4M3 LUT (each tensor has ~33 distinct scale bytes): ~+2% at 32–256 tokens. Scales drop from 11.1% to ~8.6% of bytes. Going further (5.05 bits ≈ log2 33) isn't worth the unaligned unpacking.
4. Dequant straight into wgmma A registers (swap-AB) · the main structural lever for M ≥ 17
- Orientation: weights are the M side (64-row tiles of output columns) and tokens are N (8…256, padded to a multiple of 8).
wgmma.m64nNk16.bf16takes A from registers. Each consumer warpgroup decodes E2M1 × scale into BF16 fragments in registers, so dequantized weights never touch shared memory. Activations x (B operand) come in via TMA once per expert and stay in shared memory. - E2M1 × E4M3 → BF16 is exact: ≤2 + ≤4 significant bits fit BF16's 8. Fold the block scale into the conversion with one
bf16x2multiply per pair. The bit trick is ~11 integer ops per 8 values (maximenewman): shift the nibble into BF16 exponent and mantissa position withlop3, multiply by a magic 2^k × scale. Aprmt-based 16-entry LUT is an alternative to benchmark. - Dequant is amortized over all N tokens of an expert in one instruction (N up to 256). That fixes the v16/v32 instruction-bound regime, where hot experts with ~160 tokens were re-dequantized once per 16-token item.
- For M ≤ 16, use
mma.sync m16n8k16swap-AB with 4-warp CTAs (8-warp CTAs and bigger token blocks both lost here). Keep the ring-issuer warp variant at bs8.
5. Epilogues and the A→B handoff
- Interleave gate and up columns in the repack so one thread owns both for the same column. SwiGLU is fused into the phase-A epilogue, and the BF16 intermediate goes to global (small, L2-resident).
- Phase-A completion is tracked per expert with a counter (data flag), not a grid barrier. Phase-B CTAs start streaming their w2 tiles into the ring before waiting on the flag, since weights don't depend on h. This hides the handoff dip; the ramp is paid once, not twice.
- Phase-B epilogue: y × (1+post_norm_w), and accumulate the per-(token, expert) sum of squares in shared memory, then one atomic per row. Final combine: the last CTA per token (self-resetting counter) computes Σ_e g·y_e·rsqrt(ms_e+ε) into BF16. Keep SGLang's BF16 rounding points.
6. Small-M specifics (bs1, bs8)
- bs1 is ≤8 local experts, pure GEMV. Spread phase A over all SMs by N-slices (no split-K is needed: K=4096 fits one unit). The first stage size matters more than ring depth: issue a small first chunk (4–8 KB) so math starts before the 36 KB unit lands, ~4 µs after launch.
- Read
topk_idswith the very first instruction. Every other prologue load is independent and issued in parallel.
Per-M builds
| M | MMA | Tile (weights × tokens) | Split | Notes |
|---|---|---|---|---|
| 1 | CUDA-core FMA or mma.sync m16n8 | 64 × 8 (7 padded) | N-slices across all SMs | Fastest first byte; tiny first stage |
| 8 | mma.sync m16n8k16 | 64 × 8 | stream-K | Ring-issuer warp variant |
| 16, 32 | mma.sync or wgmma N=16/32 | 64–128 × 16/32 | stream-K + DSMEM pairs | Benchmark both MMAs per size |
| 128, 256 | wgmma A-from-RF, 4 consumer warpgroups | 64 × N(tokens of the expert, ≤256) | stream-K, LPT order | 6-bit scales, KPERM3, no atomics |
| other (tests) | Nearest-larger build | — | — | Correctness only; untimed |
Dead ends (measured by others)
- Fused-slice MoE (A and B per I-slice in one CTA): 5–13% slower in all six variants. The K=16 down-projection can't share accumulators, and instruction count doubles.
- Warp-specialized GEMM with 1 producer + 12 MMA warps on an mbarrier ring: three builds landed on a ring-protocol floor equal to the old kernel (infraax).
- Dual-lane wgmma (WGS2), per-unit barriers: 8–20% slower, because they add sync to the lane that must not wait.
- L2 prefetch of MoE weights during the prior tail: ×0.99–1.00. Speculative expert prefetch, persistence windows, lookahead rings: at optimum or worse.
- Grouping a hot expert's items to share L2: no gain. Split-K at small batch with extra waiting warps: no gain.
Ideas beyond the public frontier (unmeasured, ranked)
- High. B-weight prefetch before the A→B flag (above). It hides the handoff dip, which keell0renz listed as a loss but did not remove.
- High. Dequant into A registers with N = all of an expert's tokens. This ends re-dequantization at v16/v32.
- Medium. Tail shaping: plan the stream-K so every SM's last unit is a small one. LPT plus "fill with the smallest" bin-packing cuts the 6–9 µs tail.
- Medium. Fused combine: fold the final weighted sum into the phase-B epilogue of the last-finishing expert per token. That removes the final pass at bs1/bs8.
- Low. Entropy-coded E2M1 codes. Codes are near-uniform, so the gain is small and decode is expensive.
Sources: TASK.md (spec, floors, tests); keell0renz LESSONS.md (myth, ramp and tail, 6-bit scales, stream-K, failures); alygg77 (KPERM3 gains); maximenewman (dequant op count, small-M lessons); Kushagrabainsla (atomics cost); lee101 (heaviest-first, red.release); georgxzh (SGLang µs per shape). Links on the overview.