muse_bf16_proj
Best public 1.588× · closing #1 1.728× · ceiling ≈1.9× · plan target ≈1.78×
Serving fit (Nov 14 revision)
| Question | Answer for serving |
|---|---|
| Is the 12-bit codec allowed? | It is lossless, so yes in principle. The encoder must run after load, outside model_loader/. |
| Memory | The cap is stock + 2 GB per GPU, which over 62 layers is about 32 MB of duplicated weights per layer. The BF16 originals are 37.7 MB (qkv_gate), 67.1 MB (pooler) and 16.8 MB (o); their 12-bit copies are 28.3, 50.3 and 12.6 MB. Both encodings of everything will not fit. Per matrix, either convert in place or keep a duplicate where it fits: o, or qkv_gate, but not the pooler. |
| Cost of converting in place | 12-bit decode at m ≥ 128 measured 2.4–12 µs slower, and prefill runs these GEMMs at thousands of tokens. An in-place pooler would speed decode and slow prefill. Check TTFT against the 2× gate before keeping it. |
| Dispatcher | The 30-cell sweep with per-cell winners is a kernel-judge device. In serving, use two regimes by token count (GEMV at 32 and below, stock GEMM above) and no timing sweep at start-up. |
| Router | Keep exact FP32 at every batch size. The hi/lo BF16 pair was a kernel-track allowance, and a changed expert pick changes the output that the agreement gate compares. |
| Token counts between board sizes | The boards timed m = 1, 8, 16, 32, 128 and 256. Serving also sees 24, 40, 72 and so on. Check that cost rises smoothly across the switch from the GEMV to the stock GEMM. |
| Why this port scores well | Weight bytes are read once per layer whatever the number of users, so they are fixed cost, which the score amplifies at low loads (sketch). The draft model's projections may be covered by the same kernel; profile it. |
| Smallest useful version | One 12-bit split-K GEMV kernel for qkv_gate and o at 32 tokens and below, plus the readable encoder. Everything else stays on stock. |
Plan and byte allowance: Serving track, row 2.2.
Problem
out = x · Wᵀ for five matrices at decode loads (m = 1, 16, 32) and verify loads (m = 8, 128, 256): 30 shapes, each load summing its matrices.
| Matrix | W shape (N × K) | Bytes | Spec floor m=1 | SGLang m=1 | Character |
|---|---|---|---|---|---|
| router | 256 × 8192, FP32 (x FP32) | 8.4 MB | 1.75 µs | 10.53 | Latency-bound; strict tolerance |
| qkv_gate | 2304 × 8192 | 37.7 MB | 7.87 | 13.98 | Bandwidth-bound to m≈64 |
| pooler | 4096 × 8192 | 67.1 MB | 13.99 | 21.41 | Largest; bandwidth-bound |
| o | 8192 × 1024 | 16.8 MB | 3.50 | 6.78 | Short K=1024; no split-K needed |
| unpooler | only the first 1024 used at T≤8 | ≈8.4 MB at m=1 | 1.75 | 7.04 | Latency-bound at small m |
SGLang m=1 µs from georgxzh's judge run. Router at m=8 is a SGLang outlier (26.6 µs), which makes it a cheap 3–4× row.
Ceiling arithmetic (m=1)
| Matrix | BF16 at 4.3 TB/s + 1.5 µs | × | 12-bit at 4.3 TB/s + 1.5 µs | × |
|---|---|---|---|---|
| router | 3.45 | 3.05 | 2.97 | 3.55 |
| qkv_gate | 10.29 | 1.36 | 8.09 | 1.73 |
| pooler | 17.12 | 1.25 | 13.21 | 1.62 |
| unpooler | 3.45 | 2.04 | 2.97 | 2.37 |
| o | 5.41 | 1.25 | 4.43 | 1.53 |
Lever 1: lossless 12-bit BF16 codec (already proven; get it exactly right)
Trained BF16 weights use a narrow band of exponents. keell0renz's pooler GEMV at m=1 ran 13.0 µs vs the 14.0 µs BF16 byte floor.
# prepare(): per matrix (or per 64-row panel)
e = (w_bits >> 7) & 0xFF # 8-bit exponent
base = choose b maximizing count(b <= e < b+15) # 15 codes + 1 escape code
idx = clamp(e - base, 0, 15); esc = (e < base) | (e >= base+15)
plane8 = (sign << 7) | mantissa7 # 1 byte / weight, 16 B aligned loads
plane4 = where(esc, 15, idx) # nibble-packed, 0.5 byte / weight
escapes = sorted list of (row, k, true_bf16) per CTA tile # usually << 0.1%
# kernel: w = bf16((sign,mant) | ((base+idx) << 7)); escaped positions decode as 0,
# then y[row] += x[k] * true_bf16 for that tile's escape list (exact, in FP32)
- Two planes keep loads aligned: 8-bit and 4-bit streams, each a TMA bulk copy. Decode is ~3–4 integer ops per weight (
prmt/lop3), far below the ~11 ops per weight a 13 µs pooler GEMV can afford on CUDA cores. - Store zero as a reserved code if the zero exponent falls outside the window (it often does). Otherwise every zero becomes an escape.
- Downgraded after the second pass: alygg77 measured BF16 weight entropy at 10.46 bits (1.3% escapes at 12 bits), so going below 12 bits has an upper bound of +2.4% even with free decode, realistically +0.5–1%. Also, 12-bit decode at m≥128 is 2.4–12 µs slower (integer ALU doesn't overlap HGMMA); keep it to m ≤ 32. Original idea, for reference: an entropy-coded exponent (DFloat11-style, ~11 bits per weight) with a canonical 2/3/5-bit prefix code and per-warp bit offsets. That is another ~8% fewer bytes on pooler and qkv. Risk: decode cost and bit-alignment latency at m=1. Benchmark against a bare ring first; adopt only for pooler and qkv at m ≤ 16.
- Compliant: weights must be lossless and BF16-or-smaller. The Nov 2 ban is on compressed source, so keep the encoder readable Python in
prepare().
Lever 2: kernels per regime
m ≤ 32: TMA-ring split-K GEMV ("sk2")
- A producer warp streams weight panels via
cp.async.bulkinto a 4 × 36 KB ring. Consumers decode and FMA (m ≤ 8) ormma.sync m16n8k16swap-AB (weights as M, tokens as N=8/16/32). - Stage x into shared memory once per CTA with an early bulk copy. Staging latency of 2–5 µs killed an in-kernel-reduction variant for lee101, so issue it before the first weight stage.
- The split-K reduction stays in the same launch: either a 2-CTA cluster with a DSMEM reduction, or FP32
red.global.addinto a workspace plus a last-CTA convert with a self-resetting counter (BF16 or FP16 partials are allowed since Nov 3 but measured noise-level). - Work balance for qkv (2304 rows): 4-CTA K-split clusters didn't fit all GPCs and ran in two waves (×0.87). Don't chase CTA count either. alygg77 found the qkv GEMV saturates at ~96 CTAs (10.7–11.4 µs at both 96 and 128 CTAs), and stream-K has a 0.6 µs ceiling against a 1.5–4 µs fix-up tail. Spend the effort on the per-stage decode and issue path instead.
o projection (K = 1024)
- No split-K: each CTA owns N-rows over full K. Small m: GEMV with 12-bit weights. m=128/256: custom wgmma kernels with FP32 accumulation beat cuBLAS (alygg77), plus NathenFernandes's
ol_projectionat m=128/256.
m = 128, 256: tensor-core GEMMs
- Pinned cuBLASLt plans (algorithm fixed in prepare from a sweep) as the baseline. A custom persistent stream-K wgmma kernel where tile count ≠ a multiple of 132 (pooler N=4096 → 32 × 2 tiles at 128 × 128, badly quantized on 132 SMs).
- 12-bit weights at large m: decode into shared memory with a producer warpgroup, then wgmma. Win only if the shape is still bandwidth-limited (pooler m=128 yes; m=256 marginal). Measure.
Lever 3: the router (cheapest big multiplier)
- m ≤ 8: exact FP32 FFMA GEMV (alygg77 reads the rules as requiring this). It needs 8.4 MB in ~3 µs: 256 rows × 8192 split over 128–132 CTAs, each reducing a K-slice for all 256 outputs, then a last-CTA reduction. Read x into shared memory once.
- m > 8: hi/lo BF16 on tensor cores in a single GEMM. Stack along K:
[x_hi | x_hi | x_lo] · [W_hi ; W_lo ; W_hi]gives the three significant cross terms in one FP32-accumulating GEMM with K = 3×8192. That saves lee101's separate add pass (they stacked [hi;lo] along N=512, then added). Check whether splitting x into hi/lo is within the rule. If not, keep x_hi only with [W_hi;W_lo] along K (K=16384). - m=128/256: cuBLAS on [K, 2N] plus PDL join measured 1.78× (georgxzh). Folding the add into K should take a few more µs off.
Dispatcher
Run a variant sweep for all 30 cells inside prepare(): {sk2-12bit, sk2-bf16, cluster split-K, cuBLASLt plan k, custom wgmma, NathenFernandes routertail/ol, chenwainuo rs128}. Persist the per-cell winner from judge-profile runs, not local runs; noise is ±1.5%, so require >2% to switch.
Dead ends
- Hand-written GEMV streaming weights straight into registers: register pressure from in-flight bytes cut occupancy (maximenewman). Use the shared-memory ring.
- Tensor-core GEMV (WSTREAM-style) at small m: 0.2–2.5 TB/s (infraax).
- 4-CTA K-split clusters for qkv: ×0.87 (two waves).
Sources: TASK.md via Mithun-033 (shapes, floors); georgxzh (per-shape SGLang µs); keell0renz (12-bit result, cluster failure, dispatcher); alygg77 (wgmma o-proj, exact FP32 router reading); lee101 (router stacking, x-staging lesson); maximenewman; infraax. Links on the overview.