Kimi Delta Attention: a half-life per channel
Chapter 5 gave the model one dial for how fast to forget. But a state does not hold one kind of thing. It holds the language of the document and the last number mentioned in the same matrix, and those two facts do not want the same half-life.
\(\alpha_t\) is a scalar. It multiplies the entire state, so every channel decays at an identical rate. Every step, the model must pick one memory horizon and apply it to everything it knows — and any single choice is simultaneously too forgetful for the durable facts and too retentive for the volatile ones.
It is worth being precise about how bad this is, because “one number instead of many” sounds like a minor parameterisation detail rather than a bottleneck.
Suppose a head is tracking two things: the identity of the file being edited, which should survive the whole context, and the value of the loop variable three lines up, which is stale almost immediately. The first wants \(\alpha \approx 0.9999\), a half-life of about seven thousand tokens. The second wants \(\alpha \approx 0.9\), a half-life of under seven. Those differ by three orders of magnitude. With a scalar gate the model has exactly one move available: pick something in between and be wrong about both, or oscillate \(\alpha\) token by token and corrupt whichever fact happens to be riding the wrong step. The information is separable — it lives in different directions of the state — but the forgetting mechanism is not.
Promote the scalar to a diagonal
The fix is the smallest one available. Replace the scalar \(\alpha_t\) with a vector \(a_t \in (0,1)^{d_k}\), one decay per channel, and put it on the diagonal of a matrix:
at = σ(xtWa) ∈ (0,1)dk is projected from the token, exactly like \(\alpha_t\) was, but with a \(d_k\)-wide output instead of a 1-wide one. Everything else is equation 5.1 unchanged.
Recall the shape convention: \(S \in \mathbb{R}^{d_k \times d_v}\), so row \(c\) of the state is everything ever written through key-channel \(c\). Left-multiplying by \(\mathrm{Diag}(a_t)\) scales row \(c\) by \(a_t[c]\) and nothing else. So the diagonal is not an arbitrary generalisation — it is precisely the operation that gives each key-channel its own independent forgetting rate while leaving the read and write algebra untouched.
The cost is one projection of width \(d_k\) instead of width 1. For \(d_k = 128\) that is 128 numbers per token per head instead of one — and 128 independent half-lives, from about six tokens to effectively permanent, all active at the same time.
Decay is a learned relative position bias
Worth pausing on, because it explains a design decision in the next chapter that otherwise looks careless.
A fact written \(t\) steps ago is scaled by \(a^{t}\) in each channel. That is a function of distance only — it does not depend on absolute position, and it decreases monotonically. Which is to say: exponential decay is a relative position bias, of exactly the family ALiBi made explicit, except that here it is (a) learned rather than fixed, (b) data-dependent rather than static, and (c) multi-scale by construction, because each channel carries its own exponent.
A KDA layer already encodes position, in a richer form than RoPE does. So Kimi Linear applies no positional encoding at all to its full-attention layers — NoPE. The recurrent layers supply the positional signal; the attention layers are left free to do pure content-based retrieval, which is the one thing the recurrent layers cannot do. Two mechanisms, two jobs, no redundancy.
Why a diagonal, and not any matrix?
The obvious next question is why stop here. If \(\mathrm{Diag}(a_t)\) is better than \(\alpha_t I\), why not a full \(d_k \times d_k\) transition matrix \(A_t\), and a genuinely general linear recurrence?
Because of chapter 4. The answer is not about expressiveness — a full matrix is strictly more expressive — it is about whether a parallel algorithm exists at all.
Expand equation 6.1's transition:
This is diagonal plus low rank — DPLR, the same structural family as chapters 3 and 5. And that family is exactly the set of transitions for which the chunkwise machinery works: a product of \(C\) such matrices collapses into one \(C \times C\) triangular solve, which is what made training linear in sequence length instead of sequential in it.
A general \(A_t\) has no such collapse. The product \(A_C A_{C-1}\cdots A_1\) is \(C\) dense matrix multiplies that must happen in order, and you are back to a sequential loop over the sequence — untrainable at scale, whatever its expressiveness on paper.
In this whole lineage, the space of architectures is not bounded by what is mathematically expressive. It is bounded by what admits a chunkwise parallel form. Every increment from chapter 3 forward — the delta rule, scalar decay, per-channel decay — is the largest step that stays inside diagonal-plus-low-rank. The algorithm came first and the architecture was fitted to it.
What changes in the kernel
Chapter 5 folded decay into a \(C \times C\) matrix \(G_{ij} = \gamma_i/\gamma_j\) and multiplied it elementwise into the score matrix. That worked because \(\gamma_i/\gamma_j\) was a scalar per pair of positions. It is now a \(d_k\)-vector, so there is no \(C \times C\) mask to build — naively you would need a \(C \times C \times d_k\) tensor, which for \(C = 64,\, d_k = 128\) is 500 000 elements per chunk per head, and the whole point of chunking was to avoid materialising things of that size.
The way out is to absorb the decay into the vectors instead of the scores. Write \(\gamma_i = \prod_{j \le i} a_j\) elementwise, and note that a value written at \(j\) and read at \(i\) contributes
The per-channel decay factors separate. Rescale the queries by \(\gamma\) and the keys by \(1/\gamma\), both elementwise, and the decayed attention becomes an ordinary undecayed matmul on \(\tilde q, \tilde k\). No mask tensor, no extra asymptotic cost — the same chunked algorithm from chapter 4, run on rescaled inputs.
# chapter 5 → chapter 6: what actually changes # a: (b, h, C, d) — per-channel decay, was a scalar per position g = a.cumprod(dim=-2) # γ_i, elementwise per channel (log space in practice) q_ = q * g # q̃ — absorb the decay to here... k_ = k / g # k̃ — ...and its inverse to here A = (q_ @ k_.transpose(-1,-2)).tril() # decay is already inside. no G mask.
\(k \oslash \gamma\) divides by a number that shrinks geometrically. Over a chunk of 64 positions with \(a = 0.9\), \(1/\gamma\) reaches \(10^{3}\); over 128 it reaches \(10^{6}\), and in bf16 the rescaled keys stop being representable. Production kernels therefore use secondary chunking: split each chunk into sub-chunks of 16 or 32, reset \(\gamma\) to 1 at every sub-chunk boundary, and carry the accumulated decay across boundaries in the state instead of in the vectors. The mathematics is unchanged; it is entirely a floating-point range problem, and it is the reason KDA ships as a hand-written kernel rather than a few lines of PyTorch.
The admission: some retrieval cannot be compressed
Everything so far has been an attempt to make a fixed \(d_k \times d_v\) state behave well. Chapter 2 established the hard ceiling on that project: a state of that size holds on the order of \(d\) cleanly separable associations, because you only get \(d\) mutually orthogonal directions in \(\mathbb{R}^{d}\). Gating changes which associations survive. It does not change how many.
So for a task that genuinely requires exact recall of an arbitrary token from 100 000 positions back — a key buried in a log file, a name mentioned once — no amount of gating suffices. That job needs a memory that grows with the context, which is to say it needs chapter 1.
Do not choose. Interleave. Kimi Linear runs three KDA layers for every one full-attention layer. The KDA layers carry a constant-size state and handle the bulk of the mixing at constant cost per token; every fourth layer keeps a real KV cache and does exact softmax retrieval over the entire context.
Why three-to-one, and not one-to-one or seven-to-one
The ratio is an empirical choice, but the two forces bounding it are legible.
- Pushing toward more attention: exact retrieval is not something a KDA layer can approximate its way into. If a task needs a fact that no channel happened to keep, the only recourse is a layer that still has the raw context. Too few attention layers and long-context recall collapses in the way pure linear-attention models have always collapsed.
- Pushing toward less attention: every attention layer restores an \(O(N)\) cache and \(O(N)\) bytes read per token in that layer. The savings are linear in how many layers you remove, so each one you keep is paid for at long context.
Three-to-one sits where the recall curve has flattened but the cache bill has already fallen by three quarters. It is a knee-of-the-curve decision, not a principle — and the fact that it is tunable per model is itself part of the design.
| Chapter | State per layer | Read per token | Can target one fact? | Can bulk-clear? | Timescales |
|---|---|---|---|---|---|
| 01 · softmax attention | \(O(N)\) — grows | \(O(Nd)\) | n/a — nothing is overwritten | no | exact, unbounded |
| 02 · linear attention | \(d_k\!\times\!d_v\) | \(O(d^2)\) | no | no | none — no decay at all |
| 03 · DeltaNet | \(d_k\!\times\!d_v\) | \(O(d^2)\) | yes, rank one | no | none |
| 05 · Gated DeltaNet | \(d_k\!\times\!d_v\) | \(O(d^2)\) | yes | yes, \(\alpha\to0\) | one, shared |
| 06 · KDA | \(d_k\!\times\!d_v\) | \(O(d^2)\) | yes | yes, per channel | \(d_k\), independent |
| 06 · KDA + MLA hybrid | mixed: 3 fixed, 1 growing | \(\tfrac14 O(Nd)\) | yes | yes | \(d_k\) plus exact |
Nothing, in the attention mechanism — this is roughly where the sequence-mixing story stops. But two other bills are now unpaid. First, the surviving attention layers still cache \(2 \cdot H \cdot d_h\) numbers per token, and one quarter of a large number is still a large number; that is what MLA fixes. Second, none of this addressed the feed-forward half of the block, which is where most of the parameters and most of the knowledge live — and at 2.8 trillion parameters, running a dense FFN for every token is not affordable. Chapter 7 assembles the full system, and adds one more idea that has nothing to do with tokens at all: attention along the depth axis.