Adam does not follow the gradient.

The gradient answers: which direction rises fastest per unit of distance? But "distance" means √(Δθ₁²+Δθ₂²+…) — which assumes every parameter is measured in the same units. Change that assumption and the fastest direction changes with it. Drag the anisotropy up and watch gradient descent lose the ability to make progress along the flat axis.

What "whiten" does. Substituting y′ = √κ·y turns the ellipses into circles, and one plain gradient step lands near the bottom. Nothing about the problem changed — only the ruler. Preconditioning is doing exactly this substitution without rewriting anything:

The choice of P is the choice of geometry. P = H⁻¹ is Newton's method and circularizes any quadratic — but H is n×n, so at a billion parameters that is 10¹⁸ entries. Adam takes the affordable option: P diagonal, one number per parameter, forever.

This is the honest limit of what Adam buys you. Here the same valley is rotated 32°, so the two parameters are correlated. A diagonal preconditioner can only rescale along the axes — and the axes are now the wrong frame. Adam overshoots and rings; the full metric still walks straight in.

Parameter space is bookkeeping. Two very different weight vectors can compute the same function, so a step of fixed parameter length says nothing about how much the model actually changed. Fisher information measures the step that matters: perturb θ by δ, and the model's output distribution moves by KL ≈ ½δᵀFδ.

Three ways in. As sharpness of estimation: for a Gaussian mean, F = 1/σ², and Cramér–Rao says no unbiased estimator beats variance 1/F — Fisher is literally how much the data tells you about a parameter. As a metric on function space: the version drawn above, and the one that matters for optimizers. As free curvature: for log-likelihood losses F equals the expected Hessian, so you can estimate second-order information from squared first derivatives you already computed. That last fact is what makes any of this affordable.

And the honest caveat. Adam's v is an EMA of g², and calling it Fisher requires three concessions: it keeps only the diagonal, discarding every correlation; it uses the empirical Fisher — gradients at observed labels rather than labels sampled from the model, which Kunstner et al. (2019) argue diverge badly early in training; and it divides by √v, not v, which is not what natural gradient prescribes at all. The square root is pure heuristic. "Adam approximates natural gradient" is a motivating story, not a derivation. The defensible claim is narrower: gradient scale correlates with curvature, and Adam normalizes by gradient scale.

Multiply any coordinate's gradient by a constant c and Adam's step for that coordinate is unchanged: m scales by c, √v scales by c, the ratio is identical. Below, the same problem is run at four gradient scales spanning a factor of a hundred. Gradient descent gives four different answers, one of which diverges. Adam gives one line — all four trajectories lie exactly on top of each other.

Why this is not a toy. Gradient magnitudes get rescaled constantly without anyone intending it. Mixed-precision training is the cleanest case: fp16 gradients underflow to zero, so the standard fix multiplies the loss by 1024 before backprop and divides it out after — Adam barely notices either operation. Less deliberately, magnitudes vary by orders of magnitude across depth, between weights and biases, across layers with different fan-in, and between embeddings for frequent and rare tokens. Gradient descent needs all of these roughly commensurable, which is why careful initialization, clipping, warmup, and per-layer learning rates are so load-bearing for SGD and comparatively optional for Adam.

The units also explain the learning rate. If g carries units of [loss]/[θ], then m and √v carry them too, so m̂/√v̂ is dimensionless and α carries units of [θ] — it is directly a distance in parameter space. "α = 0.001" literally means "move each weight about 0.001 per step." That is why Adam's default transfers across wildly different models and SGD's does not.

Set β₁ = 0 and ε = 0 and the update collapses to m/√v = g/|g| = sign(g) — signed gradient descent. Turn the smoothing back on and it becomes soft: a coordinate whose gradient has consistent sign saturates near ±1 and takes a full step; a coordinate that is pure noise averages toward zero and its step shrinks on its own. Three coordinates receive the same noise, and differ only in signal.

ε has units, so it sets an absolute floor. Once √v falls below it, the denominator stops tracking gradient scale and the invariance is simply gone — precisely in the small-gradient regime where you were relying on it.

Second: it is per-coordinate only. Rotate the parameter space so two coordinates mix and the diagonal cannot follow — Plate II. Natural gradient survives arbitrary smooth reparameterization; Adam survives only axis-aligned rescaling.

Third: weight decay is not invariant. An L2 penalty contributes λθ to the gradient, and that term is then divided by √v along with everything else. Parameters with large gradients receive less effective decay than parameters with small ones, so regularization strength varies silently across the network according to gradient statistics. Decoupling the decay from the adaptive denominator restores the intent — that is AdamW, and it is most of why AdamW generalizes better than Adam.

Everything so far has treated the parameters as a flat list of numbers. But a weight matrix is not a list — it is a linear map, and what matters about it is how much it amplifies its inputs. Muon changes the norm to match. Adam caps how far each entry may move. Muon caps how far the operator may move.

Writing B = UΣVᵀ, the orthogonalization throws Σ away entirely and keeps only the rotation. Five iterations of a fixed quintic polynomial approximate this in bf16 on tensor cores — no SVD anywhere. Formally it solves argmin ⟨G, O⟩ subject to ‖O‖₂ ≤ 1, whose answer is exactly UVᵀ: steepest descent under the spectral norm.

The family lines up. The Schatten-p norm is just the ℓp norm of the singular values. So SGD is steepest descent under Frobenius = Schatten-2, and Muon is steepest descent under spectral = Schatten-∞. Adam is an ℓ∞ method too — but on the entries, not the spectrum. Same shape of answer, different question about what the coordinates are.

Two things worth knowing. Muon only applies to 2D hidden weights. Embeddings, the output head, and every 1D parameter — biases, norm gains — still run AdamW. It is a supplement, not a replacement. And it keeps one momentum buffer where AdamW keeps two, which halves optimizer state on the parameters it covers.

Two readings of that question, and both have clean answers.

First: the size of the step itself. Every singular value of O equals one, so ‖ΔW‖₂ = η exactly — on every matrix, at every step, guaranteed rather than approached. Adam gives you nothing that tight. Per entry, an A×B semi-orthogonal matrix has RMS 1/√max(A,B), so RMS(ΔW) = η/√max(A,B). That is why Moonshot rescales the update by 0.2·√max(A,B), pinning the per-entry RMS at 0.2η to match what AdamW delivers. That rescaling, plus decoupled weight decay, were the two changes that made Muon work at scale without re-tuning.

Second: the speedup. The headline number is 2×. The carefully-controlled number is closer to 1.1×. Both are real measurements — they differ in how well the AdamW baseline was tuned, and at what scale.

Moonshot's scaling-law study reported roughly 2× compute efficiency, Muon matching AdamW at about 52% of the FLOPs, and trained Moonlight — a 3B/16B MoE — on 5.7T tokens to back it. Wen et al. then swept hyperparameters separately for each of eleven optimizers and found nothing exceeded 1.4× against a properly tuned AdamW, attributing the gap to the baseline's low learning rate; independent replication by Essential AI landed at 1.1–1.2× token efficiency.

The trend matters more than any single number. The advantage shrinks with scale — 1.4× at 0.1B down to 1.1× at 1.2B — and the fitted law extrapolates to Muon being marginally worse than AdamW at 7B in the 1× Chinchilla regime. That is an extrapolation past the largest model they trained, and should be read as one. The advantage is also regime-dependent: Muon leads at 1–4× Chinchilla but Soap and Kron overtake it in overtrained settings at 8× and beyond.

What survives all the caveats. Matrix-based preconditioners consistently beat entry-wise ones. That is the finding that generalizes, and it is the thesis of this entire atlas: the geometry you impose is a real choice, and for a matrix, the entries were never the right coordinates. In production, Muon or its variants now train Kimi K2, GLM-4.5, and INTELLECT-3, and PyTorch ships it natively.