muse_attention_decode
Best public 2.717× · closing #1 3.339× · ceiling ≈3.3–3.6× · plan target ≈3.45×
Serving fit (Nov 14 revision)
| Design element on this page | Under the serving rule "nothing below stock" |
|---|---|
| Query rounded to FP8 E4M3 | Stock's FA3 path does this. Fine. |
| FP16 accumulation of QKᵀ and P·V; FP16 PV with converted V | A kernel-track allowance from Nov 3. Assume out unless RULES.md says otherwise, and match stock's accumulation precision. |
Emulated exp2, __sincosf RoPE, FP8-stored P | The same: approximation beyond stock. Assume out. |
| Static-shift softmax (my hypothesis) | Mathematically exact, but it changes rounding. Experimental, and only behind the agreement gate. |
| Balanced split-KV, bytes in flight on gathers, FP32 bulk-reduce merge, one launch on the sliding path, 64-row GQA tiles | Scheduling and data movement, not numerics. These are the parts to keep. |
| Page size 1 with arbitrary rows (every key a 128 B gather) | A board condition, not a serving one. Page size is a launch flag. If it is open, test 16 or 64 tokens per page with a bulk-read path. The only layout experiment in the public ledgers was in-kernel page sorting, which lost. The cost is coarser prefix matching. See untested experiments. |
| Per-shape dispatch (five kernels) | Reduce to sliding versus full and decode versus verify, chosen by layer type and q_len. Not by batch 1/16/32. |
Also check before porting: the verify block length and page size in the launch flags (the boards used 8 tokens and page size 1), context lengths up to 1M, and the byte cost. Three paths are the most code on this site for the smallest share of the step. Give it about 90 KB at most, or leave it out. Plan: Serving track, row 2.3.
Problem
For each request and each new token at position p (one decode token, or an 8-token causal verify block):
- Split the fused projection into q, k, v and gate. RMSNorm q (then ×5.473) and k. Apply RoPE on sliding layers only.
- Write the new k and v to the FP8 E4M3 cache at the given locations.
- Compute softmax(q·Kᵀ/8)·V over the 2,048-token window (sliding) or the whole ~80K context (full).
- Output RMSNorm(attn) × sigmoid(gate) → BF16 [B, L, 16·64].
16 query heads and 2 KV heads per GPU, head_dim 64, so 8 query heads share each KV head. One cached token is 2 heads × 64 × (K+V) = 256 B. Page size 1 in scored shapes means arbitrary row locations: every key is a 128 B gather. Tolerance: ≤ 2.5× the reference's own FP8-query error + 1.5%, clamped to 3–15%. New cache entries within 8%. No skipping keys.
Where the score is (SGLang on the judge, µs)
| Load | Sliding (w=46) SGLang / floor | Full (w=16) SGLang / floor | Load µs (SGLang) | Target sliding / full | Target ratio |
|---|---|---|---|---|---|
| decode b1 | 20.9 / 0.1 | 31.3 / 4.3 | 1,462 | 4.0 / 7.0 | 4.94× |
| decode b16 | 26.6 / 1.6 | 201.1 / 63.2 | 4,441 | 5.5 / 72 | 3.16× |
| decode b32 | 31.7 / 3.3 | 384.1 / 126.4 | 7,604 | 7.0 / 140 | 2.97× |
| verify b1 | 22.9 / 0.1 | 36.6 / 4.3 | 1,639 | 5.0 / 9.0 | 4.38× |
| verify b16 | 29.1 / 1.9 | 205.6 / 63.4 | 4,629 | 7.0 / 76 | 3.01× |
| verify b32 | 34.2 / 3.7 | 389.8 / 126.8 | 7,810 | 9.5 / 148 | 2.78× |
| Geomean target | 3.46× | ||||
SGLang times are from georgxzh's final judge run. Most teams' weakest load was full verify at b16/b32 (1.0–1.2× over FA3). Closing that is the largest single gap to #1.
Path 1: sliding window (all batches, decode and verify) · latency kernel
Structure: exactly one launch per call
- Grid = (request, KV head, key chunk). The chunks of one (request, head) form a thread-block cluster. Chunk = 128–256 keys, so batch 1 gets 16–32 CTAs and the per-CTA serial chain stays short.
- Each CTA normalizes and rotates its own q (8 heads × L tokens; tiny) in registers. Q/K prep is fused; no prep kernel.
- The CTA holding the newest positions writes the new FP8 K/V to the cache and uses them straight from registers. No read-after-write through global.
- Merge: push partial (m, l, O) into the leader CTA's shared memory over DSMEM, then one cluster barrier. Remote stores go before the barrier, local reads after. The leader applies the 1/l scale, RMSNorm over 64 and the sigmoid gate, and stores BF16.
Attack the dependent-read chain (≈2 µs cold)
t0 read seq_len[b] ─┐ (cold DRAM ~0.7–1 µs)
‖ read q/k/v/gate row, │ issue ALL independent loads before waiting on seq_len:
‖ norm weights, cos/sin │ qkv row, gate, RMSNorm weights, RoPE table entry for p
t1 read page_table[win] ─┤ window = [len-2048, len): every thread issues its 8 int32 loads at once
t2 gather K/V rows (128 B) ─┘ cp.async 16 B × 8 lanes per row, all rows of the chunk in flight
t3 QKᵀ → softmax → PV → DSMEM push → cluster barrier → norm·gate → store
- Load scalars first and pin them with
asm volatile("" :: "r"(x))so nvcc can't sink them past the gathers. - For q_len 1, compute the RoPE angle in registers (
__sincosfis within BF16 accuracy, allowed since Nov 3) instead of a cold table read. Hypothesis: saves one DRAM trip on the critical path. - Warm cold code: idle warps run the merge and epilogue once with stores masked off while the gathers are in flight. That turned a 6,900-cycle cold merge into ~2,300 for Kushagrabainsla.
Math
- Round q to FP8 E4M3 (allowed). QKᵀ runs FP8×FP8 on the tensor core with no K conversion. Swap A and B: keys are the M side (64 rows), and the 8 query heads × L tokens are N (8 or 64). Decode uses
mma.sync m16n8k32 e4m3. Verify uses N = 64 = 8 heads × 8 tokens, a perfectwgmma m64n64k32. - PV: P to FP16, V FP8→FP16 (
cvt.rn.f16x2.e4m3x2), FP16 MMA with FP16 accumulation (allowed since Nov 3; keep l in FP32). - Row sums on the tensor core: multiply P by a ones column inside the PV MMA, which removes shuffles (lee101 v38).
Path 2: full-attention decode (q_len 1) · bandwidth kernel
- Persistent, balanced split-KV. Concatenate the key ranges of all (request, KV head) pairs and give each of 132 CTAs (or 264 at 2 per SM) an equal contiguous slice. That is stream-K over keys, balanced whatever the context-length mix. Batch 1 full (80K × 2 heads) becomes ~1.2K keys per CTA.
- Bytes in flight. You need ≥ 33 KB per SM in flight for ~4.3 TB/s: a 4-stage × 16 KB cp.async ring, one producer warp issuing 128 B row gathers from the page table, prefetched one stage ahead. Read the page table itself as a vector stream (it's contiguous).
- Consumer warps: FP8 QKᵀ (N=8 query heads), softmax, FP16 PV, as in Path 1.
- Merge across CTAs that share a (request, head): write the partial (m, l, O) in FP32 (FP16 partials are now allowed but measured −1%). The last arriver per segment (self-resetting counter) merges and applies norm × gate. A TMA bulk-reduction merge (
cp.reduce.async.bulk .add.f32) removes the serial merge loop (Madhavan113, used in keell0renz). An old one-at-a-time merge cost 126 dependent L2 round trips (maximenewman). - Time budget: a full-decode kernel streaming at 4.3 TB/s takes about bytes ÷ 4.3 TB/s + a few µs. The b32 floor of 126.4 µs at 4.8 TB/s means ~607 MB (mean context ≈ 74K), which is ~141 µs at 4.3 TB/s. So the 140 µs target means running at the measured ring ceiling, with the prologue hidden under the first gathers.
Path 3: full-attention verify (q_len 8, b16/b32) · the gap to #1
Kernel design (FA3-style, specialized for 64-row GQA tiles)
- Tile: one KV head × 8 tokens × 8 query heads = M = 64 rows. S = Q·Kᵀ is
wgmma m64n{128}k32 e4m3, with Q (FP8) in registers or shared memory and K tile [128 keys × 64] K-major as stored. No transpose for K. - PV: FP8 wgmma needs K-major B, and V is stored N-major. Two options: (a) the producer warpgroup converts V to FP16 while copying into shared memory, then FP16 wgmma with transpose-B (allowed for 16-bit types), with P as an FP16 A-from-registers operand and FP16 accumulation; (b) FP8 P × FP8 V with an in-shared-memory byte transpose (FA3 FP8 path). Pick (a) first: no permute work, and tensor-core FLOPs aren't the limit (≈8.4 GFLOP total at b32).
- Warp specialization: 1 producer warpgroup (page-table walk, 128 B gathers via cp.async into an mbarrier ring, V convert) and 2 consumer warpgroups ping-ponging, so one's softmax overlaps the other's MMA. Give each KV head its own consumer warpgroup and never CTA-barrier between them (lee101's diagnosis). Only mbarriers per stage.
- Split-KV over ~132–264 CTAs, the same balanced stream-K as Path 2, with the same bulk-reduce merge.
- exp budget: b32 needs ~328M exps ≈ 86 µs of MUFU at 16/clk/SM, against ~150 µs of bytes. It overlaps only if softmax is spread over both consumer warpgroups. Try FA4-style emulated exp2 on the FMA pipe for ~25–40% of elements (allowed: accurate to BF16). alygg77 profiled the v16f consumer loop as exp 40.5%, QK + V transpose 17.5%, PV barrier 12.7% and data waits 12.5%, so exp is binding on v16f and v32f. It regressed 12–21% where exp wasn't binding, so build it into the full-verify kernels only. Keep 2 CTAs per SM (1 per SM measured 16–18% slower) within the ~104-register budget (72 accumulator + 32 score + 8 P). Balancing split ends didn't change SM finish time, and the V transpose is already hidden behind QK latency.
- Causality: within the 8-token block, key j is visible to token i iff j ≤ p_i. Only the last tile of the context needs the mask, so peel it.
Lever to test: static-shift softmax (my hypothesis, not yet measured)
q and k are RMS-normalized, so logits have a hard bound computable in prepare() from the norm weights alone: |logit| ≤ 5.473/8 · 64 · max|w_q|·max|w_k| = B (RoPE is a rotation, so the bound survives). If 2B ≲ 80, use a fixed max m = B for every key. Softmax is shift-invariant, so this is exact up to FP32 rounding. In FP32, e^(s−B) stays above 1.2e-38 for any s ≥ −B.
- Removes the running-max update, the per-tile O rescale and the max exchange in merges. Split-KV merges become pure sums (bulk-reduce add of O and l), so there is no last-CTA math beyond 1/l.
- Caveat: P must stay FP32 (or be rescaled by a per-tile power of two) before the FP16 PV, because e^(−2B) underflows FP16. Keep a per-tile 2^k exponent, which is cheap: one integer add per tile, no FMAs on O.
- Guard: compute B in prepare. If the weights violate it, dispatch the normal online-softmax build. The tests where keys within a head share a direction (huge-magnitude logits) are exactly where this must be validated.
Shape dispatch summary
| Shape | Kernel | Notes |
|---|---|---|
| decode / verify, sliding, any batch | Path 1 cluster kernel | Chunk 128 at b1, 256 at b16/b32. Verify uses wgmma N=64. |
| decode full b1 | Path 2, 132–264 CTAs | Prologue diet matters most (7 µs target). |
| decode full b16/b32 | Path 2 persistent | Pure bandwidth: 4.2+ TB/s on gathers. |
| verify full b1 | Path 3 with 132 splits | About 9 µs target. |
| verify full b16/b32 | Path 3 persistent, ping-pong, optional emulated exp2 at b32 | Target ≈ 76 / 148 µs (2.7× / 2.6× FA3). |
Validation checklist
- All 16 required tests: contexts 10…1,048,576, page size 64, KV scale 1/64, DP and TP8 head layouts, edge distributions, keys at window edges.
- Per-SM timeline for b1 sliding: the time from
t0to the first gather landing should be ≤ 2 DRAM latencies. - Bare-gather skeleton at b32 full: measure the achievable rate for 128 B row gathers before tuning the math.
- Port check: the same kernels must run in the decode layer (attention wins carried over at −4…−8% on verify shapes for infraax) and in SGLang serving.
Sources: TASK.md (spec, floors, tolerances); georgxzh (per-shape SGLang µs); Kushagrabainsla (cluster/DSMEM design, cold-code lesson); lee101 (ones-wgmma row sums, CTA-barrier diagnosis); maximenewman (merge, FP8 q, block-diagonal Q); infraax (exp2 split regression, tile-size win); keell0renz (bulk-reduce merge, ZhimaoL verify kernels). Links on the overview.