Kimi's answer to the quadratic wall is to compress the past into a fixed-size state. GLM's answer is the opposite: keep the full cache, exact and lossless, and put a cheap little model in front of it whose only job is deciding which few thousand tokens are worth attending to. Nothing is blurred. Things are skipped.
Look at a trained model's attention weights over a long document and almost all of the mass sits on a handful of places: the first few tokens, the recent window, and a scattering of genuinely relevant positions far back. The other 99% of the softmax is computed, normalised, multiplied by values, and contributes nearly nothing.
So the sparse-attention bet is: predict where the mass will land, cheaply, and only compute there. Keep exact softmax over the survivors — no approximation of the attention operation itself, just a smaller argument list.
The same 56-token causal matrix, twice. Dense computes the whole lower triangle. Sparse computes eight cells per row — and the pattern it picks is not arbitrary: attention sinks at the start, a local window, and a couple of far-off anchors.
Before sparsity, GLM inherits DeepSeek's Multi-head Latent Attention. Rather than caching a key and value per head, cache a single low-rank latent vector per token and reconstruct the heads on the fly:
Only ct is stored, plus one small decoupled key that carries the rotary position information — the up-projections fold into the query side at inference, so the heads are never materialised in cache. In DeepSeek-V3's configuration that is 576 numbers per token per layer instead of 32,768.
MLA doesn't reduce attention compute at all — the triangle is still there. It reduces what you have to hold in memory, which is what makes a million-token cache survivable in the first place.
DeepSeek Sparse Attention runs in two stages, and the split is the whole trick: something very cheap scores everything, then something expensive runs on almost nothing.
Note what didn't change. The softmax is exact. The values are the real values. If the indexer picks the right 2,048 tokens, the output is nearly identical to full attention — and if it picks wrong, the model has simply not seen something, which is a different and more legible failure than a blurred memory.
The indexer has to be taught what full attention would have looked at. In DeepSeek's recipe it is trained to match the dense model's attention distribution before the model is fine-tuned to run sparse. The selector isn't a heuristic bolted on at inference — it is a learned component, and its quality is the ceiling on the whole scheme.
Forty-four tokens of history, one query at the right. The indexer scores everything, the threshold drops to the k-th best, and only the survivors get a beam. Move k and watch the needle at position 9 fall out of the selection.
Position 9 is the needle — semantically relevant, far away, and invisible to any fixed sliding window. It survives only because the indexer scored it, which is exactly the case a local-window scheme gets wrong.
The main attention term stops growing with context — past k tokens, adding history costs nothing. But the indexer term is still linear per token, which means quadratic over a sequence. It is smaller by the ratio of head dimensions, not by an order of complexity. At a million tokens the indexer is no longer a rounding error; it becomes the dominant term, which is precisely why the newer work is about making the indexer itself cheaper.
Log–log, arbitrary units, indexer head dimension taken as ~1.8% of the main head dimension. The flat cyan stretch is the win; the upward bend at the right is the indexer reasserting itself.
| GLM · sparse | Kimi · linear (KDA) | |
|---|---|---|
| the past is | kept in full, compressed by MLA | folded into a fixed-size matrix |
| memory / token | still grows — O(L) | constant |
| attention compute | O(k) + indexer | O(1) |
| failure mode | indexer misses a token | state blurs old detail together |
| recall of a hit | exact | approximate |
| hard engineering | top-k kernels, training the indexer | chunkwise kernels for a diagonal-plus-low-rank recurrence |
The GLM-5 report is worth reading on this directly: the team continual-trained a 9B baseline into several efficient-attention variants, including a linearised one, and published the comparison. Naively interleaved sliding-window attention collapsed on long-context benchmarks; searching for which layers keep full attention recovered most of the gap. They shipped sparse. MiniMax ran a similar bake-off, found hybrids looked fine on standard benchmarks but degraded on multi-hop reasoning at scale, and shipped full attention for M2 before moving to sparse for M3.
Kimi K3 is the case for the other branch, at a scale nobody had tried. Neither side has produced a result that settles it — which is why both stacks are still being built.