muse_prefill_layer

Best public 1.913× · closing #1 2.147× · plan target ≈2.3× · not part of the overall rank, but the TTFT half of serving

Serving fit (Nov 14 revision)

Low priority for the score; watch it for the TTFT gate. Arithmetic from the judge's own times: a cached 753-token turn costs about 70 ms of stock layer-kernel prefill (46 × 1,057 + 16 × 1,314 µs). Its 220 output tokens cost 220 ÷ accepted-tokens-per-step verify steps of about 16.6 ms each, which is 0.6–0.9 s at an assumed 4–6 accepted tokens. Prefill is then roughly a tenth of a turn or less. A 16K-token first-turn chunk is about 1.1–1.2 s of layer time, paid once per conversation.

Problem and loads

The same layer as decode, at agentic-coding prefill shapes. Attention is causal within the block. Sliding layers see 2,048 tokens; full layers see all context. Three loads, geomean, each load equal:

LoadShapeWeightFloorSGLangTarget
Turn 1b1, ctx 80,000, q_len 753, sliding462571,057470
b1, ctx 80,000, q_len 753, full164971,314560
Turn 4b4, ctx 80,000, q_len 753 (3,012 tokens), sliding461,0363,2171,400
same, full161,9924,6262,050
First turn16,384 tokens, ctx 0, full165,51818,0707,000
16,384, ctx 32,768, sliding1385,66317,0406,800
16,384, ctx 32,768, full167,71020,6308,800
16,384, ctx 65,536, full169,84823,50010,500
Load ratios: 2.28× · 2.28× · 2.46× → geomean2.34×

SGLang µs from georgxzh's judge run. Floors are max(bytes at 4.8 TB/s, FLOPs at 989 TFLOP/s), so they assume BF16 attention math. FP8 queries are legal, so the attention part of the floor can be undercut. The full-layer targets above sit only ~10% over the BF16 floor for that reason.

Where the time goes (my estimates)

ShapeAttention GFLOPBF16 timeFP8 QKᵀ + FP16 PVAll-FP8 (if FP8 P is allowed)Other big costs
Turn 1 full248251 µs188 µs125 µsMoE ~950 local routes, bandwidth-bound (~450 MB ≈ 105 µs); dense ~140 GFLOP
Turn 1 sliding66 µs5 µs3 µsSame MoE + dense: MoE-dominated
First turn, ctx 0550556 µs417 µs278 µsDense ~3.2 TFLOP; MoE ~1 TFLOP (both compute-bound)
First turn, ctx 64K4,9485.0 ms3.75 ms2.5 msAttention-dominated
Three regimes: (1) T=753/3,012 sliding: MoE weight bandwidth plus Marlin inefficiency. SGLang's Marlin path barely moved for most teams (≈1.0×). (2) Full layers at 80K context: attention FLOPs and KV re-reads. (3) 16K first turns: compute-bound dense and MoE GEMMs, with long-context attention on top.

Design

1. Attention: FP8, GQA-packed, split-KV

2. QKV GEMM with a fused epilogue

3. o-proj → residual → norm → pooler ∥ router in one GEMM

4. MoE: one persistent grouped kernel, two orientations

5. Unpooler + residual 2 + final norm

6. Dense GEMMs at 16K

Known bugs and tests

Expected-value ranking

  1. High: Marlin → persistent swap-AB MoE on turn loads (turn 1/4 sliding sit at ≈1.0× for most teams; this is where 2× comes from).
  2. Medium (downgraded): FP8 QKᵀ GQA-packed attention. alygg77 found attention is only 12.3% of the first-turn load, and a kernel with no exp still took 87–91% of the time. It matters mainly on turn-1/turn-4 full shapes.
  3. High (new): fuse the ~1.1 ms of memory-bound glue (boundaries 768 µs, gather 123, combine 117, route 67, qk prep 33; 13.5% of the load) into GEMM epilogues and prologues. Hiding glue under GEMMs lost (GEMMs are power-capped at ~1.53 GHz and slow 8–15% with co-running traffic).
  4. High (new): the expert kernel at T=753 runs at 423 TFLOP/s with ~25% padding and is instruction-bound. Cut dequant instructions per k-block (−12 gave +5.8%) and padding.
  5. Medium: fused QKV epilogue; pooler+router GEMM.
  6. Medium: fused-dequant MoE vs pre-pass + DeepGEMM at 16K.
  7. Low: overlapping attention and MoE across token chunks. Both are tensor-bound, so it only fills wave tails.

Sources: TASK.md (shapes, floors, tests, router rule); georgxzh (SGLang µs, DeepGEMM path); lee101 (KV re-read diagnosis); keell0renz (pinned cuBLASLt, ZhimaoL FMHA, fresh-thread bug). Links on the overview.