muse_bf16_proj

Best public 1.588× · closing #1 1.728× · ceiling ≈1.9× · plan target ≈1.78×

Serving fit (Nov 14 revision)

The second kernel port, after the experts. The five m=1 projections sum to 59.7 µs of the 171.3 µs stock sliding layer at decode b1, about a third. In an optimized layer the dense GEMVs still sat 16 µs above their floor (51.5 against 34.9 µs, alygg77), which is about 1 ms per 62-layer step.
QuestionAnswer for serving
Is the 12-bit codec allowed?It is lossless, so yes in principle. The encoder must run after load, outside model_loader/.
MemoryThe cap is stock + 2 GB per GPU, which over 62 layers is about 32 MB of duplicated weights per layer. The BF16 originals are 37.7 MB (qkv_gate), 67.1 MB (pooler) and 16.8 MB (o); their 12-bit copies are 28.3, 50.3 and 12.6 MB. Both encodings of everything will not fit. Per matrix, either convert in place or keep a duplicate where it fits: o, or qkv_gate, but not the pooler.
Cost of converting in place12-bit decode at m ≥ 128 measured 2.4–12 µs slower, and prefill runs these GEMMs at thousands of tokens. An in-place pooler would speed decode and slow prefill. Check TTFT against the 2× gate before keeping it.
DispatcherThe 30-cell sweep with per-cell winners is a kernel-judge device. In serving, use two regimes by token count (GEMV at 32 and below, stock GEMM above) and no timing sweep at start-up.
RouterKeep exact FP32 at every batch size. The hi/lo BF16 pair was a kernel-track allowance, and a changed expert pick changes the output that the agreement gate compares.
Token counts between board sizesThe boards timed m = 1, 8, 16, 32, 128 and 256. Serving also sees 24, 40, 72 and so on. Check that cost rises smoothly across the switch from the GEMV to the stock GEMM.
Why this port scores wellWeight bytes are read once per layer whatever the number of users, so they are fixed cost, which the score amplifies at low loads (sketch). The draft model's projections may be covered by the same kernel; profile it.
Smallest useful versionOne 12-bit split-K GEMV kernel for qkv_gate and o at 32 tokens and below, plus the readable encoder. Everything else stays on stock.

Plan and byte allowance: Serving track, row 2.2.

Problem

out = x · Wᵀ for five matrices at decode loads (m = 1, 16, 32) and verify loads (m = 8, 128, 256): 30 shapes, each load summing its matrices.

MatrixW shape (N × K)BytesSpec floor m=1SGLang m=1Character
router256 × 8192, FP32 (x FP32)8.4 MB1.75 µs10.53Latency-bound; strict tolerance
qkv_gate2304 × 819237.7 MB7.8713.98Bandwidth-bound to m≈64
pooler4096 × 819267.1 MB13.9921.41Largest; bandwidth-bound
o8192 × 102416.8 MB3.506.78Short K=1024; no split-K needed
unpooleronly the first 1024 used at T≤8≈8.4 MB at m=11.757.04Latency-bound at small m

SGLang m=1 µs from georgxzh's judge run. Router at m=8 is a SGLang outlier (26.6 µs), which makes it a cheap 3–4× row.

Ceiling arithmetic (m=1)

MatrixBF16 at 4.3 TB/s + 1.5 µs×12-bit at 4.3 TB/s + 1.5 µs×
router3.453.052.973.55
qkv_gate10.291.368.091.73
pooler17.121.2513.211.62
unpooler3.452.042.972.37
o5.411.254.431.53
Implication: at small m, the score is set by bytes (so compress losslessly) and by fixed latency (one launch, no memset, no second reduction kernel). At m=128/256 it is set by tensor-core efficiency and wave quantization. The pooler at m=256 is ~17 GFLOP ≈ 17 µs vs ~16 µs of bytes. Different kernels; dispatch per (matrix, m).

Lever 1: lossless 12-bit BF16 codec (already proven; get it exactly right)

Trained BF16 weights use a narrow band of exponents. keell0renz's pooler GEMV at m=1 ran 13.0 µs vs the 14.0 µs BF16 byte floor.

# prepare(): per matrix (or per 64-row panel)
e = (w_bits >> 7) & 0xFF                     # 8-bit exponent
base = choose b maximizing count(b <= e < b+15)  # 15 codes + 1 escape code
idx  = clamp(e - base, 0, 15); esc = (e < base) | (e >= base+15)
plane8  = (sign << 7) | mantissa7              # 1 byte / weight, 16 B aligned loads
plane4  = where(esc, 15, idx)                  # nibble-packed, 0.5 byte / weight
escapes = sorted list of (row, k, true_bf16) per CTA tile   # usually << 0.1%
# kernel: w = bf16((sign,mant) | ((base+idx) << 7)); escaped positions decode as 0,
#         then y[row] += x[k] * true_bf16 for that tile's escape list (exact, in FP32)

Lever 2: kernels per regime

m ≤ 32: TMA-ring split-K GEMV ("sk2")

o projection (K = 1024)

m = 128, 256: tensor-core GEMMs

Lever 3: the router (cheapest big multiplier)

Dispatcher

Run a variant sweep for all 30 cells inside prepare(): {sk2-12bit, sk2-bf16, cluster split-K, cuBLASLt plan k, custom wgmma, NathenFernandes routertail/ol, chenwainuo rs128}. Persist the per-cell winner from judge-profile runs, not local runs; noise is ±1.5%, so require >2% to switch.

Dead ends

Sources: TASK.md via Mithun-033 (shapes, floors); georgxzh (per-shape SGLang µs); keell0renz (12-bit result, cluster failure, dispatcher); alygg77 (wgmma o-proj, exact FP32 router reading); lee101 (router stacking, x-staging lesson); maximenewman; infraax. Links on the overview.