Learning to forget on purpose

The delta rule deletes an association only as a side effect of writing a replacement. But a model that finishes a document and starts a new one does not have replacements for the thousand facts it no longer needs — it needs a way to say "release that capacity" with no new content at all.

And the geometry says this cannot be patched with the delta rule alone. \(I - \beta k^{\top}k\) contracts along one direction and leaves the other \(d-1\) untouched. Clearing a \(d\)-dimensional state would take \(d\) separate delta writes — 128 tokens spent doing nothing but erasing. That is not an eviction policy, that is a garbage-collection pause.

The other primitive

There is a much blunter operation available, and it is the entire contribution of gated linear attention and Mamba-2. If we were still doing chapter 2's purely additive write, adding forgetting would take one character:

S_old = cache
S_new = k @ v
# cache = S_old + S_new                 ← chapter 2
cache = alpha * S_old + S_new          ← decay the past, add the new at full strength

With \(\alpha_t \in (0,1)\) predicted from the token, the state becomes a leaky accumulator. Two consequences fall straight out.

The state is now provably bounded. Under a constant \(\alpha\) with unit-norm writes, \(\|S_t\| \le \sum_{j\ge 0}\alpha^{j} = 1/(1-\alpha)\). Without decay, an additive state's norm drifts upward roughly as \(\sqrt N\) and there is nothing to stop it. This is the framing Mamba-2 used: decay first, then add at full strength, so the state cannot grow without bound.

Every association acquires a memory horizon. A fact written \(t\) steps ago survives with weight \(\alpha^{t}\), so its half-life in tokens is

Neither primitive can do the other's job

The two mechanisms fail in exactly complementary ways, and stating why is the argument for combining them.

The gated delta rule

Put both transforms on the state. The decay scales everything; the Householder term removes the one direction being rewritten:

α_t ∈ (0,1) and β_t ∈ (0,1) are both data-dependent scalars projected from the token. The mathematics is otherwise identical to chapter 3.

The corners of the \((\alpha,\beta)\) square recover every model in the series so far:

Making decay survive chunking

Everything in chapter 4 assumed that moving a value from position \(j\) to position \(i\) inside a chunk left it unchanged. With decay it does not: it gets multiplied by every \(\alpha\) in between. Define the running product from the chunk's start,

A cumulative product is the multiplicative analogue of a prefix sum — and in practice it is computed as one, in log space, because \(\gamma\) underflows fp16 quickly: \(\log\gamma_i = \sum_{j\le i}\log\alpha_j\), so the ratio \(\gamma_i/\gamma_j\) is a difference of two prefix sums, exponentiated at the last moment.

Then decay enters chapter 4's algorithm in exactly three places, and nowhere else:

1. The key–key products inside \(T\) pick up \(\gamma_i/\gamma_j\), because \(u_j\) has decayed on its way to position \(i\).
2. The query reading the incoming state is scaled by \(\gamma_i\), the decay accumulated from the chunk boundary to this position.
3. The state handed to the next chunk is scaled by \(\gamma_C\), the chunk's total decay, and each key's contribution by \(\gamma_C/\gamma_i\).

# the diff against chapter 4, schematically
g  = alpha.cumprod(-1)                       # γ_i : decay from chunk start (done in log space)
Gm = g[:, :, None] / g[:, None, :]            # γ_i / γ_j : pairwise decay, a C×C matrix

K_beta = K * beta[..., None]
V_beta = V * beta[..., None]
T = -((K_beta @ K.transpose(-1,-2)) * Gm).tril(-1)   ← 1. pairwise decay
for i in range(1, C): T[i,:i] += (T[i,:,None] * T[:,:i]).sum(-2)
T += torch.eye(C)
W = T @ (K_beta * g[..., None])
U = T @ V_beta

for i in range(L // C):
    u_i     = U[i] - W[i] @ S
    o_inter = (q_i * g_i) @ S                      ← 2. query decays from chunk start
    A_i     = ((q_i @ k_i.t()) * Gm).tril()
    o_intra = A_i @ u_i
    S       = g_C * S + (k_i * (g_C / g_i)).t() @ u_i   ← 3. state carries chunk decay
    O[i]    = o_intra + o_inter

The asymptotic cost is unchanged. One extra \(C\times C\) elementwise multiply, one prefix sum of length \(C\), and one extra scalar projection per token. Gated DeltaNet is not a more expensive model than DeltaNet in any way that shows up in a profile.