Inside the Transformer Decoder

STAGE 01

From text to vectors

A decoder never sees words. A tokenizer first splits text into a sequence of integer IDs $t_1, t_2, \dots, t_n$ drawn from a vocabulary $\mathcal{V}$ (typically 32k–256k entries). Each ID indexes a row of a learned embedding matrix, giving every position its starting vector:

$$x_i^{(0)} \;=\; W_E[t_i] \;\in\; \mathbb{R}^{d},\qquad W_E \in \mathbb{R}^{|\mathcal{V}|\times d}$$ d = d_model, e.g. 768 (GPT-2 small) … 16 384 (frontier models)

At this instant the sequence is just a matrix $X^{(0)} \in \mathbb{R}^{n\times d}$: one row per token, and every row is context-free — the vector for ball is identical wherever it appears. Two problems must now be solved: the model needs to know where each token sits, and each token's vector must come to reflect what surrounds it.

Position: rotate, don't add

Classic transformers added a learned or sinusoidal position vector, $x_i^{(0)} = W_E[t_i] + p_i$. Modern decoders mostly use RoPE (rotary position embedding): inside attention, queries and keys are rotated in 2-D planes by an angle proportional to their position,

$$\tilde{q}_m = R_{\Theta,m}\,q_m,\qquad \tilde{k}_n = R_{\Theta,n}\,k_n,\qquad R_{\Theta,m}=\mathrm{blockdiag}\!\Big(\begin{bmatrix}\cos m\theta_j & -\sin m\theta_j\\ \sin m\theta_j & \cos m\theta_j\end{bmatrix}\Big)_{j=1}^{d_k/2}$$

with per-plane frequencies $\theta_j = 10000^{-2j/d_k}$. Because rotations compose, the score between positions $m$ and $n$ satisfies $\tilde{q}_m^{\top}\tilde{k}_n = q_m^{\top} R_{\Theta,\,n-m}\, k_n$: attention sees only the relative offset $n-m$, which is why RoPE generalizes across sequence lengths far better than absolute positions.

STAGE 02

The residual stream

The cleanest way to read a decoder is not "layer feeds layer" but as a residual stream: each token keeps a running vector $x_i \in \mathbb{R}^d$, and every sublayer merely reads from it, computes something, and adds the result back. With pre-normalization (the modern default), one layer $\ell$ is:

$$X \;\leftarrow\; X + \mathrm{Attn}\big(\mathrm{Norm}(X)\big) \qquad\text{then}\qquad X \;\leftarrow\; X + \mathrm{MLP}\big(\mathrm{Norm}(X)\big)$$ two residual additions per layer · repeated L times (L ≈ 12 – 120+)

Three consequences fall out of this additive design:

Identity by default. A sublayer that outputs $0$ leaves the stream untouched, so gradients flow unimpeded through $L$ layers — the $+x$ path is a gradient highway: $\frac{\partial (x + f(x))}{\partial x} = I + \frac{\partial f}{\partial x}$.
A shared bus. Anything layer 3 writes into token $i$'s vector is readable by layers 4…$L$. Layers communicate through the stream, not around it.
Strict locality — except attention. Norms, MLPs, and the final unembedding all act on one row at a time. The only operation in the entire network whose output for token $i$ depends on token $j\neq i$ is self-attention.

Normalization, precisely

Before each sublayer, every token vector is rescaled independently. Most current models use RMSNorm:

$$\mathrm{RMSNorm}(x) \;=\; \frac{x}{\sqrt{\tfrac{1}{d}\sum_{j=1}^{d} x_j^{2} \,+\, \varepsilon}} \;\odot\; g, \qquad g \in \mathbb{R}^{d} \text{ learned}$$ acts per token, per layer — no information crosses lanes here

This keeps the stream's magnitude controlled even as dozens of sublayers keep adding to it (in practice the raw residual norm still grows steadily with depth — normalization happens on the read, not on the stream itself).

FIG. 1Anatomy of one decoder layer (pre-norm)

In the blue band, lanes exchange information (arcs always flow left→right: past→future). In the amber band, each lane is transformed alone by the same MLP weights. Both bands add their output back into the lane via +.

STAGE 03

Self-attention: how tokens share information

Here is the core question: token $i$ has a vector; tokens $1,\dots,i$ have vectors; how should $i$ pull in what it needs? Attention answers with a content-addressed, differentiable lookup. Each token emits three linear projections of its (normalized) stream vector:

$$q_i = W_Q\,x_i,\qquad k_j = W_K\,x_j,\qquad v_j = W_V\,x_j \qquad W_Q,W_K,W_V \in \mathbb{R}^{d_k\times d}$$ query: “what am I looking for?” · key: “what do I advertise?” · value: “what do I hand over if matched?”

Compatibility between a query and every key is a scaled dot product, with a causal mask that forbids looking at the future:

$$s_{ij} \;=\; \frac{q_i \cdot k_j}{\sqrt{d_k}} \;+\; M_{ij}, \qquad M_{ij}=\begin{cases}0 & j \le i\\[2pt] -\infty & j > i\end{cases}$$

The $\sqrt{d_k}$ matters: if $q,k$ have roughly unit-variance i.i.d. coordinates, then $\mathrm{Var}(q\cdot k)=d_k$; dividing by $\sqrt{d_k}$ keeps scores $\mathcal{O}(1)$ so softmax doesn't saturate and gradients survive. The mask's $-\infty$ becomes exactly $0$ after softmax — future tokens don't just get small weight, they get none.

Scores become weights by a row-wise softmax, and token $i$'s update is the weighted sum of value vectors:

$$\alpha_{ij} = \frac{e^{s_{ij}}}{\sum_{j'\le i} e^{s_{ij'}}}, \qquad o_i = \sum_{j\le i} \alpha_{ij}\, v_j, \qquad \Delta x_i = W_O\, o_i$$

Because $\alpha_{i,\cdot}$ is a probability distribution ($\alpha_{ij}\ge 0,\ \sum_j \alpha_{ij}=1$), $o_i$ is a convex combination of value vectors — token $i$'s update literally lives inside the convex hull of what earlier tokens offer. In one matrix form over the whole sequence:

$$\mathrm{Attn}(X) \;=\; \underbrace{\mathrm{softmax}\!\Big(\frac{QK^{\top}}{\sqrt{d_k}} + M\Big)}_{\text{who talks to whom }(n\times n)}\;\underbrace{V}_{\text{what gets said}}\;W_O^{\top}$$ the QK circuit decides the routing; the OV circuit decides the payload — two separable jobs

FIG. 2The causal mask — who is allowed to see whom

Hover a cell.

Rows = query token i (reading).
Cols = key token j (being read).

Lower triangle (blue): allowed, score kept. Upper triangle (gray): $s_{ij} \to -\infty$, so $\alpha_{ij}=0$ exactly. This is why the same network can be trained on all $n$ next-token predictions of a sequence in parallel — each position honestly never saw its own future.

FIG. 3 — INTERACTIVEOne attention head, one sentence — tap any token to make it the query

Arc thickness and bar length are the softmax weights $\alpha_{ij}$; the mono numbers on the right show the pre-softmax score $s_{ij}=q_i\!\cdot\!k_j/\sqrt{d_k}$ feeding each weight. Try it — the head splits its mass between robot and ball, resolving a pronoun with nothing but dot products.

Multi-head: many conversations at once

A single softmax pattern is one routing scheme. Real layers run $h$ heads in parallel, each with its own small $W_Q^{(h)},W_K^{(h)},W_V^{(h)}$ acting in a subspace of size $d_k = d/h$:

$$\mathrm{MHA}(X) = \big[\,\mathrm{head}_1 \,\|\, \mathrm{head}_2 \,\|\, \cdots \,\|\, \mathrm{head}_h\,\big]\, W_O, \qquad \mathrm{head}_r = \mathrm{softmax}\!\Big(\tfrac{Q_r K_r^{\top}}{\sqrt{d_k}}+M\Big)V_r$$ e.g. d = 4096, h = 32 → each head reads/writes a 128-dim slice; GQA/MQA variants share K,V across heads to shrink the cache

Because each head owns an independent $n\times n$ pattern, one layer can simultaneously copy the previous token, track a pronoun's referent, park excess attention on a “sink” token, and link a verb to its arguments:

FIG. 4Four heads, same layer, same sentence — four different routing programs

Patterns like these are consistently found in trained models (previous-token heads, attention sinks on token 1, induction-style copy heads, syntactic heads). The layer's output concatenates all of them, so token it receives the previous-token signal and the coreference signal in the same update.

STAGE 04

The MLP: computation inside one token

After attention has gathered context into token $i$'s vector, the second sublayer processes it — strictly per token. The same two (or three) weight matrices are applied to every position independently; no lane ever touches another here:

$$\mathrm{FFN}(x) = W_2\,\sigma\!\big(W_1 x\big) \qquad\text{or (SwiGLU)}\qquad \mathrm{FFN}(x) = W_2\big(\,\mathrm{Swish}(W_1 x)\odot W_3 x\,\big)$$ W₁ expands d → 4d (≈ 8/3·d per matrix for SwiGLU), W₂ projects back 4d → d · ~2/3 of all parameters live here

A productive way to read this block is as a giant key–value memory: each row of $W_1$ is a pattern detector (“does the vector currently encode capital-city-of?”), the nonlinearity gates which detectors fire, and the corresponding columns of $W_2$ are the vectors written back into the stream when they do (“then add Paris-flavored features”). Attention moves information; the MLP transforms and enriches it — lookup and reasoning divided cleanly between the two sublayers.

FIG. 5Position-wise expand → gate → project (one lane shown; all lanes identical & independent)

The expansion to $4d$ gives the network a huge basis of learned features to test against; the nonlinearity makes the mapping input-dependent; the projection composes the fired features back into a $d$-dim write to the residual stream.

STAGE 05

Layer by layer: what happens to an embedding

Stack $L$ of these layers and iterate. Formally, each token's trajectory through the network is:

$$x_i^{(\ell+1)} \;=\; x_i^{(\ell)} \;+\; \underbrace{\sum_{h=1}^{H} W_O^{(h)} \!\!\sum_{j\le i} \alpha^{(h)}_{ij}\, W_V^{(h)}\,\hat{x}_j^{(\ell)}}_{\text{between tokens}} \;+\; \underbrace{\mathrm{FFN}^{(\ell)}\!\big(\hat{x}_i^{(\ell+\frac12)}\big)}_{\text{within token}}$$ x̂ = normalized read of the stream · after L layers, x_i has mixed information from every path of ≤ L attention hops

Depth buys composition. One attention layer can only move information one hop: $j \to i$. Two layers can chain hops — layer 1 writes something about token $j$ into token $k$, layer 2 reads it from $k$ into $i$. After $L$ layers, the receptive field is every path of length $\le L$ through the causal graph, and later attention patterns are computed from vectors that already contain earlier attention's output — routing decisions become context-dependent in a way no single layer can express. The classic example is the induction circuit: a layer-1 previous-token head plus a layer-2 “find where this happened before and copy what came next” head, which is how models complete …A B … A → B for patterns never seen in training.

Empirically, a token's vector migrates through recognizable regimes: early layers sharpen local syntax and detokenization, middle layers build semantic and relational features, late layers rotate the vector toward the output vocabulary — from “what this token is” to “what comes next.” You can watch this with the logit lens: unembed the stream at every layer and see what it would predict.

FIG. 6 — INTERACTIVEOne token's vector across depth · prompt: “The robot picked up the ball because it was ___”

Residual stream of final token was at layer 0 — drag to move through the network

each cell = one of $d$ coordinates (schematic) · the vector is rewritten in place, layer after layer

LOGIT LENS · top next-token guesses

Early layers barely know more than bigram statistics (“the”, “a”). Around the middle, attention has delivered it → ball and the guess shifts to plausible adjectives. By the last layers the stream is essentially the answer, pre-rotated into unembedding space: heavy.

STAGE 06

Out the other end: logits and generation

After the final layer, only the last position matters for generating the next token. Its vector is normalized once more and projected against the unembedding matrix, scoring every vocabulary entry at once; a temperature-scaled softmax turns scores into a sampling distribution:

$$z = \mathrm{Norm}\big(x_n^{(L)}\big)\, W_U \in \mathbb{R}^{|\mathcal{V}|}, \qquad P(t_{n+1}=w \mid t_{1:n}) = \frac{e^{z_w/T}}{\sum_{w'} e^{z_{w'}/T}}$$ W_U is often tied to W_Eᵀ · T<1 sharpens, T>1 flattens · top-p / top-k prune the tail before sampling

Sample a token, append it, and run again — that loop is text generation. Naively this re-processes the whole prefix each step, but notice what actually changes: keys and values of past tokens are functions of past stream states, which the causal mask guarantees can never be affected by the new token. So they are computed once and stored in the KV cache:

$$o_{n+1} = \mathrm{softmax}\!\Big(\tfrac{q_{n+1}\,[k_1\cdots k_{n+1}]}{\sqrt{d_k}}\Big)\,[v_1\cdots v_{n+1}]^{\top} \qquad \text{— one new q, cached K,V}$$ per-step cost drops from O(n²d) to O(nd) · cache size = 2 · L · h_kv · n · d_k values — the real memory bill of long context

FIG. 7Cost accounting — where the FLOPs and memory go

component	acts	time (full seq)	time (1 step, cached)	parameters
attention (QKᵀ, ·V)	between tokens	O(n² · d)	O(n · d)	4d² per layer
MLP	within token	O(n · d²)	O(d²)	≈ 8d² per layer
norms	within token	O(n · d)	O(d)	d per norm
un/embedding	within token	O(n · d · \|V\|)	O(d · \|V\|)	\|V\| · d (often tied)

The quadratic term — the price of letting every token attend to every earlier token — belongs to attention alone. Everything else is linear in sequence length, which is exactly what you'd expect from the blue/amber split running through this page.

The one-sentence summary

A decoder is $n$ parallel vector pipelines, each repeatedly rewritten in place by two alternating moves: a content-addressed convex mixture over the past (attention — the only cross-token step, made honest by the causal mask), and a shared per-token feature memory (the MLP). Stack the pair $L$ times so hops compose, then read the last vector against the vocabulary. Everything else — RoPE, RMSNorm, multi-head splits, KV caches — is engineering to make those two moves stable, expressive, and fast.