The argument, in one page
Seven chapters, seven mechanisms, one sentence: a memory of fixed capacity needs an eviction policy, and a learned selective read is the best eviction policy we know how to build. Everything else was implementation.
The chain
The series has a shape that is easy to miss while you are inside it: each chapter's fix is the next chapter's problem. Not a list of improvements — a chain, where every link is forced by the one before it.
Everything, in one table
| Mechanism | The state | Update rule | Eviction policy | What it cost |
|---|---|---|---|---|
| Softmax attention 01 · GPT-2, 2019 |
KV cache, \(O(N)\), one private slot per token | append | none — nothing is ever evicted | \(O(N)\) memory, \(O(N)\) bytes read per decode step |
| Linear attention 02 · Katharopoulos 2020 |
\(S \in \mathbb{R}^{d_k\times d_v}\), fixed | \(S \mathrel{+}= k^{\top}v\) | none — writes superimpose | interference beyond \(d\) associations; weaker feature map than \(\exp\) |
| Delta rule 03 · Schlag 2021 |
same | \(S \leftarrow (I-\beta k^{\top}k)S + \beta k^{\top}v\) | rank-one, content-addressed: erase exactly what this key held | a sequential dependency that blocks parallel training |
| Chunkwise / WY 04 · Yang 2024 |
same | unchanged — this is an algorithm, not an architecture | — | one \(C\times C\) triangular solve per chunk; \(C\) chosen for tensor cores, not for FLOPs |
| Gated DeltaNet 05 · Mamba-2 + delta |
same | \(S \leftarrow \alpha(I-\beta k^{\top}k)S + \beta k^{\top}v\) | rank-one replace and full-rank release | one shared half-life for the whole state |
| KDA 06 · Kimi Linear 2025 |
same | \(S \leftarrow \mathrm{Diag}(a)(I-\beta k^{\top}k)S + \beta k^{\top}v\) | per-channel half-life; \(d_k\) independent horizons | numerical range — needs sub-chunking and a hand-written kernel |
| MLA hybrid 06 · 3 KDA : 1 attention |
mixed: 3 fixed, 1 growing but latent-compressed | both | the fixed layers evict; the attention layer does not | ≈¼ the cache traffic, and the concession that compression has a floor |
| MoE 07 · 898 experts, 18 active |
parameters, not activations | router selects top-k | selects which knowledge to load, per token | every expert must stay resident; routing crosses devices |
| AttnRes 07 · every 12 layers |
block outputs along depth | \(h_\ell = \sum_i \alpha_i f_i(h_i)\), \(\sum\alpha=1\) | selects which earlier computation to read | ≈2% latency; reported 1.25× compute advantage |
One operation, three axes
Read down the eviction column and the same shape appears nine times. A query. A set of keys. A normalised weighted read of the associated values. What differs between MLA, AttnRes and an MoE router is only what the keys are indexed by.
A fixed-capacity associative memory holds on the order of \(d\) separable facts, because there are only \(d\) orthogonal directions in \(\mathbb{R}^d\). Once it is full, a purely additive write adds interference rather than information. So capacity forces an eviction policy, and a good eviction policy has to be learned — hand-designed ones (FIFO, uniform decay, fixed windows) throw away the wrong things. Gating, routing, and the delta rule are three shapes of the same learned decision, and attention is the most effective selective read we currently have to implement it.
What is not settled
Three things in this lineage are load-bearing and not yet proven, and it is worth naming them rather than reporting the numbers as conclusions.
Kimi Linear's headline is that a hybrid outperforms full attention under controlled comparison, not merely that it is cheaper. That is a strong claim, and controlled comparisons at one scale have a poor record of surviving to the next. The efficiency argument stands on its own — fewer bytes read per token is not disputable. The quality argument should be held more loosely until it replicates independently at several scales.
Chapter 6 argued that per-channel decay is a learned, multi-scale relative position bias, which is why the attention layers can drop positional encoding entirely. That is elegant, and it is also an empirical bet. Exponential decay is monotone in distance; it cannot express “exactly 4 tokens back” the way a rotary phase can. Whether that matters appears to depend on the task in ways not yet well characterised.
Every mechanism here replaces exhaustive computation with a learned choice. Exhaustive computation is wrong loudly; a learned choice is wrong quietly. A router that consistently misroutes a rare domain, a channel whose decay was fit badly, an AttnRes query that never learns to look back — none of these produce an error, they produce a slightly worse model in a way no aggregate loss number distinguishes from “needs more training.” The field has gotten much better at building selection mechanisms than at auditing them.
Where the pressure goes next
Trace which resource was binding in each chapter and the trajectory is legible. GPT-2 was bound by cache bytes. Linear attention traded that for a bound on capacity. The delta rule and gating spent the next four chapters getting the most out of that capacity, and hit the wall that \(d\) slots is \(d\) slots. K3's answer was to stop trying: keep some exact attention, and buy the rest of the capacity as sparse parameters rather than as state.
Which relocates the binding constraint to somewhere new. A 2.8T-parameter model with 2% activation is not compute-bound and not cache-bound — it is bound by having 2.8 trillion parameters resident somewhere, and by the interconnect that moves tokens to whichever device holds the experts they routed to. That is a systems problem, not an attention problem, and it is where the next several years of this story will be written.
The other pressure is the quiet one from the box above. Seven chapters of increasingly sophisticated learned selection have made model behaviour depend on choices that are hard to observe and harder to verify. Building better selectors is going well. Knowing what they selected, and whether it was right, is not yet a solved discipline.