From GPT-2 to Kimi K3
Twenty-two thousand five hundred and eighty GPT-2 models fit inside one Kimi K3. Seven years, four orders of magnitude. The interesting question is not how much bigger it got — it is what stopped working at that size, and what had to be invented to fix it.
One question, asked eight times
A language model reads a sequence and has to produce the next token. To do that it needs somewhere to put what it has already read. Call that place the state.
Every architecture in this series differs from its predecessor in exactly one respect: what the state is, how it gets written, and what gets thrown out to make room. That is the whole plot. Attention, linear attention, the delta rule, gating, hybrids, mixture-of-experts, residual attention — they are not seven unrelated tricks. They are seven successive answers to one question about memory.
A memory of fixed size needs an eviction policy. Softmax attention avoids the question by never having a fixed size — it keeps everything, and pays for it. Every architecture after it accepts a fixed budget, and then has to answer: when the budget is full, what goes?
That framing is why this series uses two inks and only two. Everything the model keeps is drawn in blue. Everything it discards, or pays, is drawn in red. Where a mechanism has to decide between the two — a gate, a delta, a router — it is drawn in plum.
The chapters
GPT-2, and the price of perfect recall
The decoder-only baseline, the attention it runs, and the KV cache that makes decoding possible — and unaffordable. Why softmax attention can never forget, and why that is both its strength and its bill.
Creates the problem: state grows linearly with everything you have ever said.Linear attention: move the nonlinearity
Apply the feature map to q and k separately instead of to their product, and the whole computation re-associates into a fixed matrix. Constant memory, constant time per token. The exact cost of that trick, written out.
Fixes cache growth. Introduces: interference once you write more facts than the state has room for.The delta rule: read before you write
Fast Weight Programmers. Instead of adding a new association on top of whatever was there, first look up what is stored at this key, subtract it, and write the difference. A worked 2×2 example where linear attention gets the wrong answer and the delta rule gets it exactly right.
Fixes interference. Leaves: can only replace a fact it has a replacement for.Making it parallel: the chunkwise form
The hard chapter. A recurrence that reads its own state cannot be run as a prefix sum. The WY reparameterisation turns every correction inside a chunk into one triangular solve — after which DeltaNet is ordinary linear attention. Chunk size C turns out to be a dial between attention and an RNN.
Fixes trainability. The insight: attention and recurrence are the same computation at two settings of one knob.Learning to forget on purpose
Mamba-2 decays the whole state by a scalar each step; the delta rule edits one entry precisely. Neither can do the other's job. Gated DeltaNet does both, and the cumulative decay turns out to be a prefix sum in log space.
Fixes bulk clearing. Leaves: one forgetting rate for every channel in the state.Kimi Delta Attention: a half-life per channel
One scalar becomes a diagonal matrix. Every channel of the state gets its own decay rate, and therefore its own memory horizon — some near-permanent, some lasting ten tokens. Plus the hybrid: three KDA layers to one full-attention layer.
Fixes single-timescale memory. Admits: some retrieval must stay exact, so keep real attention every fourth layer.Kimi K3: the assembled system
Twenty-three four-layer macrocycles. Multi-head Latent Attention and why its KV cache is one vector per token. 898 experts in a compressed latent space. SiTU, and how adding a tanh made inference three times slower. And AttnRes — attention run along the depth axis instead of the token axis.
The argument, in one page
Every mechanism in the series placed in one table, and the claim they all support: a fixed-capacity associative memory needs an eviction policy, and attention is the best selective-read mechanism we have — along the token axis, the expert axis, and the depth axis alike.
Where the pressure goes next.Notation, once, for all eight chapters
Everything in this series uses one convention, chosen to match the code rather than the papers. Vectors are rows. The state is a matrix that maps a key to a value.
- \(N\) · sequence length
- How many tokens have been read. Grows during generation.
- \(d\) · head dimension
- Usually 64 or 128. Where a distinction matters, \(d_k\) is the key width and \(d_v\) the value width.
- \(q_t, k_t, v_t \in \mathbb{R}^{1\times d}\)
- Query, key and value for token \(t\), as row vectors.
- \(S_t \in \mathbb{R}^{d_k \times d_v}\) · the state
- The associative memory. Write an association with an outer product, \(S \mathrel{+}= k^{\top} v\). Read it back with a single matrix product, \(o = q\,S\). Every architecture from chapter 2 onward differs only in what sits between those two lines.
- \(C\) · chunk size
- How many tokens are processed as one tile. 64 or 128 in practice, for reasons that are entirely about GPUs (chapter 4).
- \(\beta_t \in [0,1]\) · write strength
- How hard token \(t\) overwrites what is already at its key. Chapter 3.
- \(\alpha_t\) · decay
- How much of the old state survives this step. A scalar in chapter 5, a vector in chapter 6.
Katharopoulos et al., Transformers are RNNs (2020) · Schlag, Irie & Schmidhuber, Linear Transformers are Secretly Fast Weight Programmers (2021) · Yang et al., Parallelizing Linear Transformers with the Delta Rule over Sequence Length (2024) · Dao & Gu, Mamba-2 (2024) · Yang, Kautz & Hatamizadeh, Gated Delta Networks (2025) · Kimi Linear (2025). Each chapter states what its paper actually contributed, separately from what the surrounding folklore claims.
That a transformer turns tokens into vectors, that attention involves queries, keys and values, and that matrix multiplication exists. Nothing else. Every equation is derived in place, and every piece of engineering — why chunk size is 64, why an activation function can cost 3× at inference — is explained rather than asserted.