muse_prefill_layer
Best public 1.913× · closing #1 2.147× · plan target ≈2.3× · not part of the overall rank, but the TTFT half of serving
Serving fit (Nov 14 revision)
- The gate, not the goal. TTFT must stay within 2× stock at each load. If per-user speed is measured on decode only (to confirm), faster prefill earns nothing directly; it helps only by giving GPU time back to decode at 16 and 32 users.
- It may matter more than "the gate" suggests. If TTFT is checked where your system carries extra users at stock's speed, prefill has to keep up on a busier node and caps the low-load ratios (sketch). Settle that before ranking this page last.
- Cheap levers first: chunked-prefill size and how prefill interleaves with decode steps, both through flags or a small general policy (serving plan, rows 0.1 and 1.5).
- Shared code only. The one kernel here worth its bytes is the expert kernel at 30–120 tokens per expert, and only if it is the same code path as the decode port. Stock has no sequence-parallel prefill or MegaMoE on Hopper, so Marlin runs here too.
- Rule risks: FP16 PV accumulation and FP8-stored P are kernel-track allowances, and the pooler-plus-router GEMM needs the hi/lo BF16 router, which is another. Assume all three are out.
- Regression risk from other ports: converting dense weights to 12-bit in place speeds decode GEMVs and slows prefill GEMMs. Check TTFT after every dense change.
Problem and loads
The same layer as decode, at agentic-coding prefill shapes. Attention is causal within the block. Sliding layers see 2,048 tokens; full layers see all context. Three loads, geomean, each load equal:
| Load | Shape | Weight | Floor | SGLang | Target |
|---|---|---|---|---|---|
| Turn 1 | b1, ctx 80,000, q_len 753, sliding | 46 | 257 | 1,057 | 470 |
| b1, ctx 80,000, q_len 753, full | 16 | 497 | 1,314 | 560 | |
| Turn 4 | b4, ctx 80,000, q_len 753 (3,012 tokens), sliding | 46 | 1,036 | 3,217 | 1,400 |
| same, full | 16 | 1,992 | 4,626 | 2,050 | |
| First turn | 16,384 tokens, ctx 0, full | 16 | 5,518 | 18,070 | 7,000 |
| 16,384, ctx 32,768, sliding | 138 | 5,663 | 17,040 | 6,800 | |
| 16,384, ctx 32,768, full | 16 | 7,710 | 20,630 | 8,800 | |
| 16,384, ctx 65,536, full | 16 | 9,848 | 23,500 | 10,500 | |
| Load ratios: 2.28× · 2.28× · 2.46× → geomean | 2.34× | ||||
SGLang µs from georgxzh's judge run. Floors are max(bytes at 4.8 TB/s, FLOPs at 989 TFLOP/s), so they assume BF16 attention math. FP8 queries are legal, so the attention part of the floor can be undercut. The full-layer targets above sit only ~10% over the BF16 floor for that reason.
Where the time goes (my estimates)
| Shape | Attention GFLOP | BF16 time | FP8 QKᵀ + FP16 PV | All-FP8 (if FP8 P is allowed) | Other big costs |
|---|---|---|---|---|---|
| Turn 1 full | 248 | 251 µs | 188 µs | 125 µs | MoE ~950 local routes, bandwidth-bound (~450 MB ≈ 105 µs); dense ~140 GFLOP |
| Turn 1 sliding | 6 | 6 µs | 5 µs | 3 µs | Same MoE + dense: MoE-dominated |
| First turn, ctx 0 | 550 | 556 µs | 417 µs | 278 µs | Dense ~3.2 TFLOP; MoE ~1 TFLOP (both compute-bound) |
| First turn, ctx 64K | 4,948 | 5.0 ms | 3.75 ms | 2.5 ms | Attention-dominated |
Design
1. Attention: FP8, GQA-packed, split-KV
- Pack query heads into M. 8 query heads share each KV head, so a CTA tile of 16 tokens × 8 heads = 128 rows streams one KV tile once for all of them. lee101's turn-load attention reused 8-token verify blocks and re-read KV ~94× at T=753. FA-style 64–128-row q-tiles are the fix.
- QKᵀ in FP8 (q rounded to E4M3 is allowed; K is FP8 in cache, K-major: no transpose). PV in FP16 with V converted while staging (transpose-B legal for 16-bit wgmma) and FP16 accumulation (allowed since Nov 3). If FP8 P is confirmed legal, use FA3's FP8 path with an in-shared-memory V transpose for another ~1.5×.
- T=753 gives 48 q-tiles × 2 KV heads = 96 CTAs, under 132 SMs. Add split-KV ×2–3 on full layers (80K keys) with a bulk-reduce merge. 16K first turns: 1,024 q-tiles; persistent scheduler with causal-aware LPT (longest rows first).
- Warp specialization as FA3: producer warpgroup (paged gathers; page size 1 means 128 B rows, so gather into shared memory with cp.async), two consumer warpgroups ping-ponging softmax and MMA. Emulated exp2 for part of the elements only if the timeline shows MUFU-bound (likely at FP8 rates).
- Baseline to beat: ZhimaoL's FMHA with the "N72 PV layout" in keell0renz's file.
2. QKV GEMM with a fused epilogue
- Arrange qkv_gate output columns so a 64-column N-tile is exactly one head. The epilogue does q/k RMSNorm (64-wide, in-register), the ×5.473 scale, RoPE (sliding), the FP8 cache write for k and v, and BF16 stores of q and gate. That removes a full read-and-write pass over [T, 2304] plus a launch.
3. o-proj → residual → norm → pooler ∥ router in one GEMM
- Above 8 tokens the router may run as a BF16 hi/lo pair with FP32 accumulation. Append W_router,hi and W_router,lo as 512 extra N-columns of the pooler GEMM (K = 8192 shared). One GEMM gives pooler output plus router logits (hi+lo summed in the epilogue). That saves a launch and a full read of h. Check router-tolerance margins on the outlier-distribution test.
- The residual and rmsnorm boundaries are row-wise: emit sum of squares from GEMM epilogues and apply the scale in the consumer's prologue.
4. MoE: one persistent grouped kernel, two orientations
- T=753/3,012 (≈30–120 tokens per expert, hot up to a few hundred): weight-bandwidth bound. Use the decode MoE's swap-AB wgmma with dequant into A registers (weights as M, tokens as N ≤ 256 per instruction), stream-K, LPT. This replaces Marlin, where most teams were stuck at ≈1.0×.
- 16K tokens (hundreds to thousands of tokens per expert): compute-bound BF16. Keep swap-AB with A-from-registers. Dequant happens once per 64 × 16 weight fragment and is amortized over N=256 tokens, with no shared-memory write of dequantized weights. Alternative: a dequant pre-pass to BF16 plus DeepGEMM grouped BF16 (georgxzh: 1.30–1.55× on first turns). Benchmark both. The pre-pass costs ~1.6 GB of extra traffic (≈0.4 ms), so the fused path should win.
- Token gather: route-sort in-kernel (as decode MoE). Gather x rows (8 KB each) by TMA bulk copies into the B tile. SwiGLU in the A epilogue; per-expert RMSNorm and the weighted combine in the B epilogue and a last-arriver pass.
5. Unpooler + residual 2 + final norm
- Pinned cuBLASLt or custom wgmma with a fused epilogue: α2·s + β2·rmsnorm(f·(1+w)) needs a full-row reduction. Use the two-pass sum-of-squares trick (epilogue partials, then a tiny row kernel), or N-tile = 8192 per CTA-cluster with a DSMEM reduction.
6. Dense GEMMs at 16K
- Pinned cuBLASLt plans (keell0renz's baseline), or a persistent wgmma with stream-K and the fused epilogues above. Target ≥750 TFLOP/s sustained. Watch the clock drop (~1.81 GHz under MMA load).
Known bugs and tests
- keell0renz's file mis-routes "turn 1, full" when state is first used from a fresh OS thread on the default stream. Bind the context per thread and re-create TMA descriptors per thread.
- All 5 pre-timing tests: q_len 300 at ctx 0, ctx 3000 full, batch 2/3 mixed, an outlier distribution, and 4,096 at ctx 4,096 with page size 64 in one test.
- 15-minute cap: the 16K shapes are long, so keep variant sweeps in prepare small for this problem.
Expected-value ranking
- High: Marlin → persistent swap-AB MoE on turn loads (turn 1/4 sliding sit at ≈1.0× for most teams; this is where 2× comes from).
- Medium (downgraded): FP8 QKᵀ GQA-packed attention. alygg77 found attention is only 12.3% of the first-turn load, and a kernel with no exp still took 87–91% of the time. It matters mainly on turn-1/turn-4 full shapes.
- High (new): fuse the ~1.1 ms of memory-bound glue (boundaries 768 µs, gather 123, combine 117, route 67, qk prep 33; 13.5% of the load) into GEMM epilogues and prologues. Hiding glue under GEMMs lost (GEMMs are power-capped at ~1.53 GHz and slow 8–15% with co-running traffic).
- High (new): the expert kernel at T=753 runs at 423 TFLOP/s with ~25% padding and is instruction-bound. Cut dequant instructions per k-block (−12 gave +5.8%) and padding.
- Medium: fused QKV epilogue; pooler+router GEMM.
- Medium: fused-dequant MoE vs pre-pass + DeepGEMM at 16K.
- Low: overlapping attention and MoE across token chunks. Both are tensor-bound, so it only fills wave tails.
Sources: TASK.md (shapes, floors, tests, router rule); georgxzh (SGLang µs, DeepGEMM path); lee101 (KV re-read diagnosis); keell0renz (pinned cuBLASLt, ZhimaoL FMHA, fresh-thread bug). Links on the overview.