Softmax attention keeps every token it has ever seen and re-reads all of them for each new one. Kimi Delta Attention keeps a fixed-size matrix instead, and spends its cleverness on deciding what to overwrite. This page builds it from scratch — the state, the write rule, and the one change that separates KDA from what came before.
Standard attention computes a weighted average over the entire past, with weights recomputed from scratch for each new token:
Two bills arrive. Compute: the sum runs over t terms, so a full sequence costs O(L²). Memory: to produce token t+1 you must still be holding every ks and vs — the KV cache — and it grows without bound. At a million tokens that cache is the thing that breaks, not the FLOPs.
Linear attention refuses the second bill outright. Instead of a list that grows, it keeps one matrix of fixed size and folds each new token into it.
Left: the KV cache, one column per token, forever. Right: the state matrix S, 8×8 here, rewritten in place. Both are watching the same token stream.
Drop the softmax and the sum collapses. Write St for a matrix of shape dv×dk that accumulates outer products, and reading becomes a single matrix–vector product:
That is a key–value store written with addition. If the keys were exactly orthogonal and unit length, reading with q = ki would return vi cleanly. They are not, so every read returns the value you asked for plus a smear of everything else. Cost per token is now constant: one outer product in, one product out, no cache.
And here is the catch that the last five years of research has been about. S never gets smaller. Add a thousand tokens and you have a thousand overlapping outer products fighting for the same 64 numbers. Pure linear attention doesn't forget — it blurs.
The delta rule replaces blind addition with a correction. Before writing, look up what is currently stored at this key, and only write the difference:
The clean way to see this is as one step of gradient descent, performed at inference time. Give the state a loss that says reading at kt should return vt:
So βt is a learning rate the network predicts per token. At β=0 the token is ignored; at β=1 whatever was stored at that key is fully replaced. The model decides, token by token, how hard to write.
The same 8×8 state at three moments. Reading at k gives the value already there; the erase term subtracts it; the write term puts v in its place. Only the row and column touched by k change.
The delta rule only clears the key you are writing to. Everything else sits there indefinitely. Gated DeltaNet adds a decay term — before each write, shrink the entire state by a scalar the model predicts:
This works, and it forces a choice the model cannot escape. α near 1 keeps everything, including the stale garbage. α at 0.99 halves the memory every 69 tokens, which is fine for local syntax and fatal for a fact you read 40,000 tokens ago. One number decides the fate of all 64 cells at once.
KDA's change is small to write down. Replace the scalar with a vector — one forget rate per key dimension:
Now the state has a spectrum of timescales inside it. Some channels are scratch space that turns over every few tokens; others hold on for tens of thousands. The model routes what it wants to keep into the slow channels and lets the fast ones churn. A single retention curve becomes eight.
The horizon of channel i — how long until a write there decays to 1/e — falls straight out of the gate:
A scalar gate keeps the state transition cheap to invert and to parallelise. A diagonal gate makes it a diagonal-plus-low-rank transition, which is more expressive and much harder to turn into a fast GPU kernel. Most of the Kimi Linear paper is that kernel, not the equation above.
Eight channels, eight faders. A fact is written into all of them at t = 0. Move the time marker and watch what survives. The dashed line is what a single scalar gate would do if it were set to the average of your eight — it is the curve KDA is allowed to escape.
Each fader sets one channel's horizon τ. Push two faders to the right and drop the rest: that is a model choosing to remember one thing for a long time while the rest of the state stays useful for local work.
Written as above, the update is sequential: you cannot compute St without St−1. That is death on hardware built for large matrix multiplications. The standard trick, and the one KDA uses, is chunking.
The result stays linear in sequence length while spending nearly all its time in the operations the hardware is fastest at. This is why linear attention became practical around 2024–2025 and not in 2020, when the same equations already existed. The maths was never the bottleneck; the kernel was.
A fixed-size state is lossy by construction, and the loss shows up exactly where you would fear: exact recall of a specific token far back. So KDA is not used alone. Kimi Linear interleaves three KDA layers with one full-attention layer, repeating up the stack, and Kimi K3 is described as the same shape — mostly KDA with periodic full attention.
Green layers are full attention and hold a KV cache. Amber layers are KDA and hold a fixed-size state instead. Three quarters of the cache simply stops existing.
One elegant consequence: in Kimi Linear the full-attention layers use no positional encoding at all. The KDA layers are recurrent, so they already know where they are — position is implicit in the order of the updates. The global layers can be left position-agnostic and just do retrieval. Removing RoPE from the layers that handle long-range lookup is a plausible reason long-context behaviour holds up.
The division of labour is the whole design: KDA carries the running summary cheaply, and one layer in four is allowed to go back and read the actual tokens.
| property | full softmax | KDA (linear, per layer) |
|---|---|---|
| memory / token | grows forever | constant — one matrix |
| compute | O(L²) | O(L) |
| exact recall | lossless, any distance | lossy — competes for the same cells |
| fails on | the memory bill | multi-hop chains over long spans |
This is where the field genuinely disagrees. MiniMax built a hybrid linear model, then reverted to full attention for M2, reporting that hybrids matched full attention on standard benchmarks but developed clear deficits in complex multi-hop reasoning once scaled — and that the hard part was measuring the loss at all, not designing the architecture. Zhipu, building GLM-5, ran the same comparison and also went the other way, choosing sparse attention over linearising.
Kimi K3 is the counter-argument, and currently the strongest one: a 2.8T model with a KDA-dominant stack that landed at the top of LMArena's frontend coding board on release. Whether the fixed-size state costs something the benchmarks aren't catching is the open question, and the weights are the way to find out.