Kimi's answer to the quadratic wall is to compress the past into a fixed-size state. GLM's answer is the opposite: keep the full cache, exact and lossless, and put a cheap little model in front of it whose only job is deciding which few thousand tokens are worth attending to. Nothing is blurred. Things are skipped.
Look at a trained model's attention weights over a long document and almost all of the mass sits in a handful of places: the first few tokens, the recent window, and a scattering of genuinely relevant positions far back. The other 99% of the softmax is computed, normalised, multiplied by values, and contributes nearly nothing.
So the sparse-attention bet is: predict where the mass will land, cheaply, and only compute there. Keep exact softmax over the survivors — no approximation of the attention operation itself, just a shorter argument list.
The same 56-token causal matrix, twice. Dense computes the whole lower triangle. Sparse computes eight cells per row — and the pattern it picks is not arbitrary: attention sinks at the start, a local window, and a couple of far-off anchors.
Before sparsity, GLM inherits DeepSeek's Multi-head Latent Attention. Rather than caching a key and value per head, cache a single low-rank latent vector per token and reconstruct the heads on the fly:
The obvious objection is that you have traded memory for compute — surely you now have to reconstruct all those heads on every step? You do not, and the reason is a small algebraic trick worth seeing explicitly. Attention only ever uses the key through an inner product with the query:
Push WUK onto the query side and it is absorbed into the query projection. You score directly against the cached latents. The same manoeuvre works on the output side with WUV. So only ct is stored, plus one small decoupled key carrying the rotary position information, and the per-head keys and values are never written to memory at all.
There is a second consequence that matters here: because every query head scores against the same cached latent, MLA behaves like multi-query attention at read time. One gather serves all heads — which is exactly the property sparse attention needs, since gathering a scattered top-k is a memory-access problem before it is a FLOP problem.
Log-scaled bars. MLA doesn't reduce attention compute at all — the triangle is still there. It reduces what you have to hold and move, which is what makes a million-token cache survivable in the first place.
DeepSeek Sparse Attention runs in two stages, and the split is the whole trick: something very cheap scores everything, then something expensive runs on almost nothing.
Note what didn't change. The softmax is exact. The values are the real values. If the indexer picks the right 2,048 tokens, the output is nearly identical to full attention — and if it picks wrong, the model has simply not seen something, which is a different and more legible failure than a blurred memory.
Note also the ReLU, which is doing more work than it looks. It clips negative similarities to zero rather than letting them cancel positives, so the score is a sum of evidence-for rather than a signed balance. That makes the ranking better behaved in low precision, which is what lets the whole thing run in FP8.
Forty-four tokens of history, one query at the right. The indexer scores everything, the threshold drops to the k-th best, and only the survivors get a beam. Move k and watch the needle at position 9 fall out of the selection.
Position 9 is the needle — semantically relevant, far away, and invisible to any fixed sliding window. It survives only because the indexer scored it, which is exactly the case a local-window scheme gets wrong.
Everything above assumes the indexer's ranking is any good. Nothing about the architecture guarantees that — a randomly initialised scorer would select 2,048 arbitrary tokens and destroy the model. So the selector is not a heuristic bolted on at inference. It is trained, and the recipe is a two-stage continued pre-training that GLM adopts wholesale from DeepSeek, precisely to avoid the astronomical cost of training a sparse model from scratch.
The violet sliver on the left is the 2.1B-token dense warm-up — about 0.2% of the budget. Teaching the indexer what to imitate is cheap. Teaching the model to trust it is not.
This is the most important structural fact about sparse attention and the easiest to skim past. The quality ceiling of the whole scheme is the indexer's ranking, and that ranking is a distillation of dense attention. Sparse attention does not discover which tokens matter — it learns to predict which tokens a dense model would have cared about, and then saves the cost of asking.
Because the mechanism is bolted onto a trained dense model, a lab can convert an existing checkpoint rather than committing to a new architecture at pre-training time. That is a much smaller bet than linearising, where the attention mechanism has to be chosen before the expensive run begins. Some of the preference for sparse over linear among Chinese labs is an economics argument, not purely a quality one.
The main attention term stops growing with context — past k tokens, adding history costs nothing there. But the indexer term is still linear per token, which means quadratic over a sequence. It is smaller by the ratio of head dimensions, not by an order of complexity. At a million tokens the indexer is no longer a rounding error; it becomes the dominant term, which is precisely why the newer work is about making the indexer itself cheaper.
Log–log, arbitrary units, indexer head dimension taken as ~1.8% of the main head dimension. The flat cyan stretch is the win; the upward bend at the right is the indexer reasserting itself. Drag k to see the crossover move.
Quoting a single speedup number hides that inference has two phases with opposite bottlenecks, and sparsity helps each for a different reason.
All L tokens go through at once. The machine is doing enormous matrix multiplications and is compute-bound: the GPU's arithmetic units are the constraint.
Dense attention here is genuinely O(L²) in FLOPs. Cutting each row to k entries cuts the arithmetic directly. This is where sparse attention gives its cleanest, most dramatic win — and why long-prompt agentic workloads benefit most.
bound by: arithmeticOne token at a time. There is barely any arithmetic to do; the machine spends its time reading the KV cache out of memory. It is bandwidth-bound.
Here sparsity helps by shrinking what you fetch — k entries instead of L. But the indexer still has to score every position, so you still touch all L keys, just with far smaller vectors. MLA compounds the win by making each of those reads tiny in the first place.
bound by: memory bandwidthThis is also where the top-k becomes an engineering problem rather than a maths one. The selected positions are scattered arbitrarily through the cache, so the kernel has to gather non-contiguous memory — the access pattern GPUs are worst at. A naive implementation can spend more time gathering than it saved by skipping. Much of the real work in shipping DSA is custom kernels that make the gather efficient, and MLA's single-latent-per-token layout is a large part of what makes it tractable.
GLM-5's technical report is unusually direct about the comparison, which makes it better evidence than any vendor benchmark chart. Two tables matter.
First, DSA against plain MLA on long-context benchmarks at 128K. If sparse attention were quietly lossy, this is where it would show:
| 128K benchmark | MLA (dense) | + DSA (sparse) |
|---|---|---|
| MQ-NIAH | 100.0 | 100.0 |
| MV-NIAH | 95.5 | 97.0 |
| SQuAD | 79.7 | 86.0 |
| HotpotQA | 66.3 | 63.0 |
Comparable or better on three of four, worse on the multi-hop one — which is a suggestive place to lose, given that multi-hop reasoning is exactly the weakness MiniMax reported for efficient attention generally. But the deficit is small and the rest holds, and Zhipu shipped it.
Second, and more interesting for the argument on the other page: they also continual-trained a 9B full-attention baseline into several other efficient-attention variants and compared. Sliding-window attention interleaved naively collapsed — a reported −30.35 on RULER at 128K against the full-attention baseline. Searching for which specific layers should keep full attention recovered most of that gap. They also tried linearising, with a strategy they call SimpleGDN designed to reuse pre-trained weights during continual training.
The takeaway is not that linear attention lost a fair fight. It is that Zhipu ran the comparison in the specific regime that mattered to them — converting an existing dense checkpoint cheaply — and sparse won that comparison. Kimi ran a different comparison, pre-training from scratch, and got a different answer.
Both these tables come from the team that shipped the winning option, at a scale (9B continual-training ablations) far below the model that shipped. Architectural deficits have a habit of appearing only at scale — that is the exact failure MiniMax described. Treat these as evidence about a decision process, not as settled fact.
| GLM · sparse | Kimi · linear (KDA) | |
|---|---|---|
| the past is | kept in full, compressed by MLA | folded into a fixed-size matrix |
| memory / token | still grows — O(L) | constant |
| attention compute | O(k) + indexer | O(d²), independent of L |
| what's dropped | tokens the indexer didn't rank | detail, blended into crosstalk |
| recall of a hit | exact | approximate |
| failure mode | a miss — legible, testable | a blur — quiet, hard to measure |
| hard engineering | gather kernels, training the indexer | chunkwise kernels for a DPLR recurrence |
| adoptable by | converting a dense checkpoint | committing before pre-training |
They are not, in the long run, mutually exclusive. Survey work increasingly frames them as complementary: linear attention as the cheap state compressor and sparse attention as the high-fidelity retrieval path, on the reasoning that they fail in different places. Several 2026 architectures already hybridise all three — MiniCPM-SALA, for instance, interleaves 25% sparse with 75% linear layers.
What is settled is that pure quadratic attention is finished at million-token scale, and that nobody trusts any single replacement enough to use it alone. Every shipping design keeps some full attention in reserve. The disagreement is only about what fills the rest.