Kimi K3: the assembled system

The sequence-mixing argument finished in chapter 6. What remains is everything else a 2.8-trillion-parameter model needs in order to exist: a cache small enough to serve, parameters cheap enough to run, activations stable enough to train in low precision — and one genuinely new idea that runs attention along depth instead of along the sequence.

The floor plan

The K3 language backbone is chapter 6's hybrid, repeated. Twenty-three macrocycles of four layers each — ninety-two layers. Inside every macrocycle, three KDA layers and one MLA layer, in that order. The first layer of the network uses a dense feed-forward block; every layer after it uses a latent mixture-of-experts.

Compared with Kimi Linear, the changes are a short list, and the rest of this chapter is that list:

• a large increase in scale — 2.8T parameters
• Gated MLA, with a LoRA-factored query projection
• latent-space MoE, 898 experts
• SiTU in place of the usual gated activation
• blockwise AttnRes every 12 layers

MLA: cache one vector, not one per head

Chapter 1 established that decoding is bound by bytes moved, not arithmetic, and that the bytes are the KV cache. Ordinary multi-head attention stores, per token per layer, a key and a value for every head:

Multi-head Latent Attention removes the factor of \(H\). Instead of caching \(H\) keys and \(H\) values, project the token once into a single low-dimensional latent and cache only that:

Every head's key and value are reconstructed from the same \(c_t\) on demand. The obvious objection is that reconstructing them costs a matmul per head per step, which sounds worse. It does not happen, because of a small algebraic trick worth seeing:

Matrix multiplication is associative, so \(W_{UK}\) can be absorbed into the query projection at load time, and \(W_{UV}\) likewise into the output projection. At inference no up-projection is ever performed: queries are pre-transformed into latent space and attend directly against the cached \(c_t\). You cache one vector per token per layer and pay nothing extra to use it.

Head counts are illustrative rather than K3's published configuration; the ratio is the point. Combined with chapter 6's hybrid — only one layer in four holds any cache at all — this is what makes a 2.8T model servable at long context.

K3 adds two refinements. Query LoRA factors the (now large) query projection through a low-rank bottleneck, cutting its parameters and activation memory during training. Gated MLA puts an elementwise gate, projected from the input, on the attention output before it re-enters the residual stream — so the model decides, per channel, how much of what it retrieved is actually admitted. That is the same idea as chapter 6's per-channel decay, applied at the exit instead of the store.

Mixture-of-experts: buy capacity, not compute

A mixture-of-experts replaces the single wide FFN with many narrow ones and a router that picks a few per token. K3 uses 898 experts: two shared experts that process every token, and 896 routed experts of which the router selects 16.

The accounting is the whole point:

The two shared experts are not a rounding detail. Without them, every routed expert has to independently learn the generic transformations that all tokens need, duplicating that capacity 896 times. Making a couple of experts unconditional lets the routed ones specialise on what is actually token-specific.

K3 then adds latent-space MoE: down-project the input into a compressed space, run the experts there, and up-project their summed output. Expert width falls, so expert FLOPs fall — roughly by half — for the same expert count.

SiTU, and a lesson about what activations cost

The standard gated activation splits its input in half and computes \(\text{SiLU}(g) \odot u\). K3 replaces it with SiTU, which wraps both halves in a scaled hyperbolic tangent:

The function \(\beta\tanh(x/\beta)\) is worth recognising: it is the identity for \(|x| \ll \beta\) and saturates smoothly at \(\pm\beta\). It is a soft clamp with a learnable ceiling, and it does nothing at all to typical activations. Its entire job is the tail.

AttnRes: attention along the depth axis

The last idea has nothing to do with tokens. It is about the residual stream — and it is the most interesting thing in K3.

In a standard transformer, layer \(\ell\) reads the sum of the embedding and every preceding layer's output:

where h₁ is the token embedding and f_i is the attention-plus-FFN block at layer \(i\). Every term enters with weight exactly 1.

Two things go wrong with that, and they are separate problems.

Problem one: the residual stream dilutes

If successive layer outputs are roughly independent with typical norm \(\sigma\), their sum grows like a random walk:

At layer 4 a block's output is half the stream. At layer 92 it is a tenth of it. For a late layer to change the model's mind about anything, it must emit an output roughly \(\sqrt{92} \approx 9.6\) times larger than an early layer would — and large late-layer outputs are precisely the thing that destabilises training at depth. The uniform residual sum quietly imposes a penalty on depth that grows with depth.

Problem two: everyone gets the same undifferentiated bus

Layer 80 receives the same aggregated blob as layer 20. If what it actually needs is the output of layer 12 — some syntactic parse, a resolved coreference — it has to dig that out of a sum in which layer 12 is one of seventy-nine equally weighted contributors. There is no addressing.

# the core of block_attn_res — N previous blocks plus the one in progress
V = torch.stack(blocks + [partial_block])            # [N+1, B, T, D]  ← values: block outputs
K = norm(V)                                          # keys: the same, normalised
logits = torch.einsum('d, n b t d -> n b t', proj.weight.squeeze(), K)
h      = torch.einsum('n b t, n b t d -> b t d', logits.softmax(0), V)
return h

Read the softmax(0) carefully: the normalisation is over dimension 0, which is the block axis. Each token, independently, produces a distribution over the depth of the network and mixes the blocks accordingly.

And note that this fixes problem one for free. Because the weights are a softmax, they sum to one, so \(h_\ell\) is a convex combination of block outputs rather than an unbounded sum. Its norm is bounded by the largest single block output, no matter how deep the network gets. The \(\sqrt{\ell}\) dilution simply does not occur.

Why every twelfth layer, and not every layer

Because the cost is real. Applying AttnRes at every layer means keeping every earlier layer's output live, and computing a distribution over an ever-growing set of them, ninety-two times. K3 instead defines a block as the accumulated sum of attention and MLP outputs across 12 decoder layers, stores that single depth-representation, and mixes only at block boundaries. Across 23 macrocycles that yields eight AttnRes blocks.

The reported trade is about 2% additional inference latency for a 1.25× compute advantage — the latter because a network whose late layers can address early representations directly needs to spend less depth re-deriving them. Coarse granularity captures most of the benefit at a fraction of the bookkeeping, which is the same argument chunk size \(C\) made in chapter 4: full generality at the finest granularity is rarely what the hardware wants.