muse_attention_decode

Best public 2.717× · closing #1 3.339× · ceiling ≈3.3–3.6× · plan target ≈3.45×

Serving fit (Nov 14 revision)

Third in line, and only the full-attention paths. Sliding attention is a small part of a served layer: 21–34 µs of a 171–732 µs stock layer, and 9.8 of 154 µs in an optimized one (alygg77). The 16 full layers are different. At 32 requests and about 80K context they add roughly 350 µs each, about 5.7 ms of a 51 ms stock verify step; at one request they add almost nothing. Port Path 2 and Path 3 only if your profile shows long contexts at 16 and 32 users.
Design element on this pageUnder the serving rule "nothing below stock"
Query rounded to FP8 E4M3Stock's FA3 path does this. Fine.
FP16 accumulation of QKᵀ and P·V; FP16 PV with converted VA kernel-track allowance from Nov 3. Assume out unless RULES.md says otherwise, and match stock's accumulation precision.
Emulated exp2, __sincosf RoPE, FP8-stored PThe same: approximation beyond stock. Assume out.
Static-shift softmax (my hypothesis)Mathematically exact, but it changes rounding. Experimental, and only behind the agreement gate.
Balanced split-KV, bytes in flight on gathers, FP32 bulk-reduce merge, one launch on the sliding path, 64-row GQA tilesScheduling and data movement, not numerics. These are the parts to keep.
Page size 1 with arbitrary rows (every key a 128 B gather)A board condition, not a serving one. Page size is a launch flag. If it is open, test 16 or 64 tokens per page with a bulk-read path. The only layout experiment in the public ledgers was in-kernel page sorting, which lost. The cost is coarser prefix matching. See untested experiments.
Per-shape dispatch (five kernels)Reduce to sliding versus full and decode versus verify, chosen by layer type and q_len. Not by batch 1/16/32.

Also check before porting: the verify block length and page size in the launch flags (the boards used 8 tokens and page size 1), context lengths up to 1M, and the byte cost. Three paths are the most code on this site for the smallest share of the step. Give it about 90 KB at most, or leave it out. Plan: Serving track, row 2.3.

Problem

For each request and each new token at position p (one decode token, or an 8-token causal verify block):

  1. Split the fused projection into q, k, v and gate. RMSNorm q (then ×5.473) and k. Apply RoPE on sliding layers only.
  2. Write the new k and v to the FP8 E4M3 cache at the given locations.
  3. Compute softmax(q·Kᵀ/8)·V over the 2,048-token window (sliding) or the whole ~80K context (full).
  4. Output RMSNorm(attn) × sigmoid(gate) → BF16 [B, L, 16·64].

16 query heads and 2 KV heads per GPU, head_dim 64, so 8 query heads share each KV head. One cached token is 2 heads × 64 × (K+V) = 256 B. Page size 1 in scored shapes means arbitrary row locations: every key is a 128 B gather. Tolerance: ≤ 2.5× the reference's own FP8-query error + 1.5%, clamped to 3–15%. New cache entries within 8%. No skipping keys.

Where the score is (SGLang on the judge, µs)

LoadSliding (w=46)
SGLang / floor
Full (w=16)
SGLang / floor
Load µs (SGLang)Target sliding / fullTarget ratio
decode b120.9 / 0.131.3 / 4.31,4624.0 / 7.04.94×
decode b1626.6 / 1.6201.1 / 63.24,4415.5 / 723.16×
decode b3231.7 / 3.3384.1 / 126.47,6047.0 / 1402.97×
verify b122.9 / 0.136.6 / 4.31,6395.0 / 9.04.38×
verify b1629.1 / 1.9205.6 / 63.44,6297.0 / 763.01×
verify b3234.2 / 3.7389.8 / 126.87,8109.5 / 1482.78×
Geomean target3.46×
Read the table this way. At batch 1, the load is ~65% sliding latency (46×21 µs). At batch 16/32 it is ~75–80% full-attention bandwidth (16×200–390 µs). That gives two different problems, solved by different kernels behind a shape dispatcher. Full shapes need ~4.2 TB/s on a 128 B gather. Sliding shapes need the whole call in ~4–9 µs after the judge subtracts 4 µs.

SGLang times are from georgxzh's final judge run. Most teams' weakest load was full verify at b16/b32 (1.0–1.2× over FA3). Closing that is the largest single gap to #1.

Path 1: sliding window (all batches, decode and verify) · latency kernel

Structure: exactly one launch per call

Attack the dependent-read chain (≈2 µs cold)

t0  read seq_len[b]           ─┐  (cold DRAM ~0.7–1 µs)
     ‖ read q/k/v/gate row,      │  issue ALL independent loads before waiting on seq_len:
     ‖ norm weights, cos/sin     │  qkv row, gate, RMSNorm weights, RoPE table entry for p
t1  read page_table[win]      ─┤  window = [len-2048, len): every thread issues its 8 int32 loads at once
t2  gather K/V rows (128 B)   ─┘  cp.async 16 B × 8 lanes per row, all rows of the chunk in flight
t3  QKᵀ → softmax → PV → DSMEM push → cluster barrier → norm·gate → store

Math

Path 2: full-attention decode (q_len 1) · bandwidth kernel

Path 3: full-attention verify (q_len 8, b16/b32) · the gap to #1

Why everyone stalled here. FA3 already runs at 205/390 µs (≈1.6× its byte floor), and custom kernels "only tied it." lee101 found its kernel CTA-barrier bound: verify full ran at 172/326 µs vs 117/210 µs for decode full over the same bytes. The fix is structural.

Kernel design (FA3-style, specialized for 64-row GQA tiles)

Lever to test: static-shift softmax (my hypothesis, not yet measured)

q and k are RMS-normalized, so logits have a hard bound computable in prepare() from the norm weights alone: |logit| ≤ 5.473/8 · 64 · max|w_q|·max|w_k| = B (RoPE is a rotation, so the bound survives). If 2B ≲ 80, use a fixed max m = B for every key. Softmax is shift-invariant, so this is exact up to FP32 rounding. In FP32, e^(s−B) stays above 1.2e-38 for any s ≥ −B.

Shape dispatch summary

ShapeKernelNotes
decode / verify, sliding, any batchPath 1 cluster kernelChunk 128 at b1, 256 at b16/b32. Verify uses wgmma N=64.
decode full b1Path 2, 132–264 CTAsPrologue diet matters most (7 µs target).
decode full b16/b32Path 2 persistentPure bandwidth: 4.2+ TB/s on gathers.
verify full b1Path 3 with 132 splitsAbout 9 µs target.
verify full b16/b32Path 3 persistent, ping-pong, optional emulated exp2 at b32Target ≈ 76 / 148 µs (2.7× / 2.6× FA3).

Validation checklist

Sources: TASK.md (spec, floors, tolerances); georgxzh (per-shape SGLang µs); Kushagrabainsla (cluster/DSMEM design, cold-code lesson); lee101 (ones-wgmma row sums, CTA-barrier diagnosis); maximenewman (merge, FP8 q, block-diagonal Q); infraax (exp2 split regression, tile-size win); keell0renz (bulk-reduce merge, ZhimaoL verify kernels). Links on the overview.