Softmax attention keeps every token it has ever seen and re-reads all of them for each new one. Kimi Delta Attention keeps a fixed-size matrix instead, and spends its cleverness on deciding what to overwrite. This page builds it from scratch — the state, the write rule, the one change that separates KDA from what came before, and the constraint that makes it fast enough to matter.
Standard attention computes a weighted average over the entire past, with weights recomputed from scratch for each new token:
Two bills arrive. Compute: the sum runs over t terms, so a full sequence costs O(L²). Memory: to produce token t+1 you must still be holding every ks and vs — the KV cache — and it grows without bound. At a million tokens that cache is what breaks, not the FLOPs.
Linear attention refuses the second bill outright. Instead of a list that grows, it keeps one matrix of fixed size and folds each new token into it.
Left: the KV cache, one column per token, forever. Right: the state matrix S, 8×8 here, rewritten in place. Both are watching the same token stream.
Drop the softmax and the sum collapses. Write St for a matrix of shape dv×dk that accumulates outer products, and reading becomes a single matrix–vector product:
That is a key–value store written with addition. Reading at q = ki gives
If the keys were exactly orthogonal the crosstalk term would vanish and this would be a lossless dictionary. They are not — in d dimensions you can only fit d mutually orthogonal directions, and there are far more than d tokens. Cost per token is now constant: one outer product in, one product out, no cache. What you buy it with is interference.
And S never gets smaller. Add a thousand tokens and you have a thousand overlapping outer products fighting for the same 64 numbers. Pure linear attention doesn't forget — it blurs.
The delta rule replaces blind addition with a correction. Before writing, look up what is currently stored at this key, and only write the difference:
The clean way to see this is as one step of gradient descent, performed at inference time. Give the state a loss that says reading at kt should return vt:
So βt is a learning rate the network predicts per token. At β=0 the token is ignored; at β=1 whatever was stored at that key is fully replaced. The model decides, token by token, how hard to write. This is the sense in which linear attention is "test-time training": the state is a tiny model being fit online, and the sequence is its training set.
The same 8×8 state at three moments. Reading at k gives the value already there; the erase term subtracts it; the write term puts v in its place. Only the directions touched by k change.
Here is the test that makes the tradeoff concrete. Write N random key–value pairs into an 8×8 state, then go back and query every key you wrote. Measure how far the returned vector is from the value you stored. Do it twice — once with plain addition, once with the delta rule.
Vertical axis is mean relative error on read-back, 0% = perfect recall. Both curves are computed live from the actual recurrences, averaged over several random draws. The vertical marker is N = d, where the keys stop being able to avoid each other.
Three things fall out of this, and they are the whole argument for and against linear attention.
Below capacity, both are near-perfect. While N < d the keys can stay close to orthogonal and the memory behaves like a real dictionary.
Past capacity, plain addition degrades immediately. Every new write adds crosstalk to every old read, and the error climbs roughly as √(N/d).
The delta rule buys you a large constant, not a different asymptote. It always writes the correction rather than the raw value, so it never double-counts a key it has seen — but it cannot manufacture storage. Widen the state and both curves shift right. Neither ever becomes lossless.
That is why the honest framing of KDA is not "attention, but cheaper." It is a lossy running summary with a controllable failure profile — and why nobody ships it without full-attention layers alongside.
The delta rule only clears the key you are writing to. Everything else sits there indefinitely, competing for the same 64 cells. Gated DeltaNet adds a decay term — before each write, shrink the entire state by a scalar the model predicts:
This works, and it forces a choice the model cannot escape. α near 1 keeps everything, including the stale garbage that is generating the crosstalk. α at 0.99 halves the memory every 69 tokens, which is fine for local syntax and fatal for a fact you read 40,000 tokens ago. One number decides the fate of all 64 cells at once.
KDA's change is small to write down. Replace the scalar with a vector — one forget rate per key dimension:
Now the state has a spectrum of timescales inside it. Some channels are scratch space that turns over every few tokens; others hold on for tens of thousands. The model routes what it wants to keep into the slow channels and lets the fast ones churn — which is also a way of managing the interference from the previous section, since retiring a channel frees the capacity it was consuming.
The horizon of channel i — how long until a write there decays to 1/e — falls straight out of the gate:
Because αt is data-dependent and applied once per step, the cumulative product ∏α encodes how far back something was written, per channel. The Kimi Linear authors describe KDA as acting like a learned, data-dependent positional encoding — which is the setup for the design choice in the stack section below, where the full-attention layers carry no positional encoding at all.
Eight channels, eight faders. A fact is written into all of them at t = 0. Move the time marker and watch what survives. The dashed line is what a single scalar gate would do if it were set to the average of your eight — it is the curve KDA is allowed to escape.
Each fader sets one channel's horizon τ. Push two faders right and drop the rest: that is a model choosing to remember one thing for a long time while the rest of the state stays useful for local work.
Written as above, the update is sequential: you cannot compute St without St−1. That is death on hardware built for large matrix multiplications — a modern GPU running one rank-1 update at a time is idle in every way that matters. The standard fix, and the one KDA uses, is chunking.
The causal triangle again. Amber blocks on the diagonal are computed exactly, as ordinary matmuls. Everything below them — the entire history before the current chunk — arrives as one matrix. Slide C and watch the two costs trade against each other.
The result stays linear in sequence length while spending nearly all its time in the operation the hardware is fastest at. This is why linear attention became practical around 2024–25 and not in 2020, when the same equations already existed. The maths was never the bottleneck; the kernel was.
It also explains the structural rhyme with the other page. Both approaches start from the same causal triangle and refuse to compute all of it. Sparse attention keeps a scattered top-k of individual cells and drops the rest. Linear attention keeps solid blocks near the diagonal and compresses the rest into a fixed-size summary. Skipping versus squashing.
Swapping a scalar for a vector costs one line of maths and an enormous amount of kernel engineering. To see why, look at the shape of the state transition. Every linear-attention variant has the form St = Mt St−1 + (write), and the cost of chunking is governed entirely by what Mt looks like when you multiply a chunk's worth of them together.
| variant | transition Mt | chunk product |
|---|---|---|
| plain linear | identity | trivial |
| scalar gate | αt·I | a cumulative product of scalars |
| delta rule | I − β k kᵀ | rank-1 updates, WY form |
| KDA | Diag(αt) − atbtᵀ | diagonal-plus-low-rank — expensive in general |
The last row is the problem. A general diagonal-plus-low-rank (DPLR) recurrence is far more expressive, but composing a chunk of them requires dividing by cumulative decay products, which goes numerically unstable and forces extra secondary chunking passes to keep the arithmetic in range. Each pass is more compute and more memory traffic. Expressiveness you cannot run at speed is not expressiveness.
KDA's answer is a constraint. Rather than learning the two low-rank vectors independently, tie both of them to the key:
Because a and b are now both functions of k, the chunk composition simplifies: the paper reports this removes two of the secondary chunking steps and roughly three matrix multiplications from the inter-chunk and output paths, arriving at close to 2× the kernel throughput of general DPLR. It also lands the update back in line with the classical delta rule rather than drifting into an arbitrary state-space model.
This is the pattern across the whole efficient-attention literature: the winning designs are not the most expressive ones, they are the most expressive ones that compose cheaply inside a chunk. Read any linear-attention paper and the equation is usually a page; the kernel is the rest.
A fixed-size state is lossy by construction — the capacity panel above is that loss, measured — and it shows up exactly where you would fear: exact recall of a specific token far back. So KDA is not used alone. Kimi Linear interleaves three KDA layers with one full-attention layer in a uniform 3:1 ratio, and Kimi K3 is described the same way: KDA-dominant with periodic full attention.
Green layers are full attention (MLA) and hold a KV cache. Amber layers are KDA and hold a fixed-size state instead. Three quarters of the cache simply stops existing.
One elegant consequence: in Kimi Linear the full-attention layers use no positional encoding at all. The KDA layers are recurrent and their cumulative gates already encode distance, so position is implicit in the order of the updates. That frees the global layers to be position-agnostic and just do retrieval — and removing RoPE from precisely the layers responsible for long-range lookup is a plausible reason the long-context behaviour holds up rather than degrading past the training length.
The division of labour is the whole design: KDA carries the running summary cheaply, and one layer in four is allowed to go back and read the actual tokens.
| Kimi Linear (research model) | Kimi K3 (flagship) | |
|---|---|---|
| params | 48B total, 3B active | 2.8T total, 16 of 896 experts |
| layer pattern | 3 KDA : 1 MLA, uniform | KDA-dominant, periodic full attention (ratio unpublished) |
| positional encoding | none on the MLA layers | unpublished |
| context | 1M | 1M |
| reported wins | ~75% KV cache cut, up to ~6× decode at 1M | ~2.5× scaling efficiency over K2 |
KDA changes how information moves along the sequence. Moonshot's second architectural change moves it along depth, and the two are described together as the reason K3 converts compute into capability more efficiently than K2.
A standard residual stream is an accumulator. Every layer adds its output to a running sum with a fixed weight of one:
In a very deep stack this is a lossy channel of its own. Something written at layer 2 has to survive being added to eighty more times before layer 82 can use it, and there is no mechanism for layer 82 to say I specifically want layer 2. Attention Residuals reframe the skip connection as a retrieval problem: let each layer attend over the outputs of all earlier layers, with learned weights.
Left: every layer hands upward with weight 1. Right: layer 6 pulls most of its input from layer 2 and almost nothing from layer 4. The full version stores every earlier layer's output, so K3 is reported to use Block AttnRes — ordinary residuals inside a block, attention over depth only between blocks — which bounds the memory cost.
The symmetry is worth pausing on. Along the sequence axis, KDA replaces "keep everything" with a compressed state and a learned gate. Along the depth axis, AttnRes replaces "add everything with weight one" with a learned selection. Both are the same instinct: an unweighted accumulator is a bad default, and a model should be allowed to choose.
| property | full softmax | KDA (linear, per layer) |
|---|---|---|
| memory / token | grows forever | constant — one matrix |
| compute / token | O(L·d) | O(d²), independent of L |
| training compute | O(L²) | O(L·C) chunked |
| exact recall | lossless, any distance | lossy — see the capacity panel |
| fails on | the memory bill | multi-hop chains over long spans |
| failure is | an out-of-memory error | a quietly wrong answer |
That last row is the uncomfortable one, and it is why the field genuinely disagrees. MiniMax built a hybrid linear model, then reverted to full attention for M2, reporting that hybrids matched full attention on standard benchmarks but developed clear deficits in complex multi-hop reasoning once scaled — and, pointedly, that the hard part was measuring the loss, not designing the architecture. A lossy state does not announce itself; it produces fluent output that is missing something. Zhipu, building GLM-5, ran its own comparison across efficient-attention variants and also went the other way, choosing sparse attention over linearising.
Kimi K3 is the counter-argument, and currently the strongest one: a 2.8T model with a KDA-dominant stack that landed at the top of LMArena's frontend coding board on release, while Moonshot itself notes it still trails the leading proprietary models on the hardest reasoning. Whether the fixed-size state costs something the benchmarks aren't catching is the open question, and the weights are the way to find out.