The idea in one minute
A dense model is one enormous generalist forced to think about everything at once. A Mixture of Experts is a hospital: a triage desk out front, and a hallway of specialists behind it.
When you walk into an emergency room, the whole hospital does not treat you. A triage nurse looks at you for ten seconds and routes you: chest pain to cardiology, broken wrist to orthopedics. The hospital's total capability is the sum of all its departments; the cost of treating you is two or three of them. Nobody would design a hospital where every doctor examines every patient — yet that is exactly what a dense transformer does. Every token, whether it's the or a line of Rust or a stanza of Persian poetry, is processed by every single parameter.
Mixture of Experts replaces the biggest block of a transformer layer — the feed-forward network (FFN) — with many parallel copies called experts, plus a tiny learned router that scores them and activates only the top few (typically 1–10) for each token. The result decouples two quantities that dense models fuse together:
Total parameters → what the model knows
Capacity. Facts, languages, coding patterns, rare knowledge. In 2026 this is measured in the hundreds of billions to trillions. You pay for it in memory.
Active parameters → what each token costs
Compute. The FLOPs actually executed per token. Frontier MoEs keep this at 3–6% of the total. You pay for it in time and electricity — which is what your API bill prices.
That decoupling is the entire trick. Everything else in this guide — routing functions, load-balancing losses, fine-grained experts, bias-corrected gates — is engineering to make the trick actually work at scale, because a naive version fails in instructive ways.
The problem: intelligence was priced per parameter
Scaling laws say bigger models are better. Dense architecture says bigger models are proportionally more expensive — for every token, forever. MoE breaks that pricing.
Empirically, language-model loss falls smoothly as parameter count \(N\) grows — the famous power-law scaling curves. So you want \(N\) huge. The problem is arithmetic. For a dense transformer, a good approximation of the cost of processing one token is:
Compute grows linearly with every parameter you add, and you pay it on every token, at training and at inference, for the life of the model. Double the knowledge, double the electricity bill of every future request. Through 2022, that arithmetic is why frontier models plateaued in the tens-of-billions range for anything that had to be served cheaply.
Now replace each FFN with \(N_E\) experts and activate \(k\) of them. Only active parameters generate FLOPs:
Concretely, in 2026 terms: DeepSeek-V4-Pro stores 1.6T parameters but activates ~49B per token. Its per-token compute is that of a ~49B dense model — about 33× less than a hypothetical dense 1.6T — while its knowledge capacity is the full 1.6T. That is why it can be sold at $0.435 per million input tokens, roughly the price bracket of mid-size dense models, while posting frontier coding scores.
The receipts: efficiency claims from real training runs
| Evidence | Year | What was measured |
|---|---|---|
| Switch Transformer vs T5-Base | 2021 | Same quality reached with ~7× less pre-training wall-clock at fixed FLOPs per token (Google) |
| GLaM (1.2T, 97B active) vs GPT-3 | 2021 | Beat GPT-3 on average across 29 NLP tasks using ~1/3 of GPT-3's training energy and ~half the inference FLOPs |
| Mixtral 8×7B vs Llama 2 70B | 2023 | Matched or beat the 70B dense model on most benchmarks with 12.9B active params — ~6× faster inference |
| DeepSeek-V2 vs dense DeepSeek 67B | 2024 | Stronger model, 42.5% lower training cost, 93.3% smaller KV cache, 5.76× generation throughput |
| DeepSeek-V3 (671B, 37B active) | 2024 | GPT-4o-class open model trained for 2.788M H800-hours ≈ $5.6M in GPU time — vs. an estimated $50–100M for GPT-4 |
| Qwen3-Next-80B-A3B vs dense Qwen3-32B | 2025 | Better performance at under 1/10 the training cost; >10× decode throughput past 32K context |
The fine print: two currencies, not one
MoE does not make the model small — it makes it cheap to run. The full expert set must live in accelerator memory (or be streamed from somewhere), so:
- VRAM footprint scales with total parameters. Mixtral 8×7B needs ~94 GB in fp16 to hold, even though each token touches only ~26 GB of it. This is MoE's tax, and §10 covers how systems pay it.
- Decode speed scales with active parameters. Generating a token is memory-bandwidth-bound: you must read the weights you use. Reading 26 GB instead of 94 GB per token is precisely where Mixtral's ~2.9× single-stream decode advantage over an equally-sized dense model comes from — and its ~6× advantage over the 70B dense model it matches in quality.
Anatomy of an MoE layer
Take a transformer block. Leave attention alone. Replace the FFN with N experts and a router. That's the whole surgery — the subtlety is all in the router.
The math, from scratch
A standard FFN computes, for token representation \(x \in \mathbb{R}^{d}\):
An MoE layer keeps \(N_E\) independent copies \(E_1,\dots,E_{N_E}\) of that structure (usually each narrower than the original) and combines them with a learned gate:
Because most \(g_i\) are exactly zero, most \(E_i(x)\) are never computed. Sparsity here is not compression or pruning after the fact — it is conditional execution, decided per token, per layer. The gate itself is almost embarrassingly small: one linear map from the token to a score per expert, followed by a top-k and a normalization:
Three details worth internalizing:
- Routing is per token and per layer. The word "ribosome" might use experts {3, 41} in layer 5 and {17, 200} in layer 40. Kimi K2 has 61 layers × 384 experts ≈ 23,000 expert instances; a single token's path through them is one of an astronomical number of possible compositions.
- The top-k is not differentiable, and training works anyway. Gradients flow through the gate values \(g_i\) of the selected experts (the softmax weights are continuous), so the router learns to reweight and, over time, reroute. Shazeer's original trick added tunable Gaussian noise to the scores — \(s_i + \varepsilon\cdot\mathrm{softplus}(W_{noise}x)_i\) — so near-tied experts occasionally swap, giving losers a chance to receive gradient. Most modern recipes drop the noise and rely on the balancing machinery of §06 instead.
- k/N is your compute dial. With 8 experts and k=2 you run 25% of expert FLOPs; DeepSeek-V3's 8-of-256 runs ~3%. The router itself costs \(d \times N_E\) multiplies — thousands of times cheaper than one expert.
Pick a token. Watch the router score all 8 experts (bars = softmax probability), select the top-k in orange, and renormalize their weights into the final gate. Expert "specialties" are illustrative — see the caveat in §01.
Thirty-five years in nine moves
MoE is older than the transformer, older than the web browser. What changed in 2024–26 is not the idea — it's that the last engineering objections finally died.
Adaptive Mixtures of Local Experts — Jacobs, Jordan, Nowlan & Hinton
The founding paper: several small networks plus a gating network, trained jointly so each expert takes ownership of a region of the input space. The entire modern recipe in embryo, three decades early.
Conditional computation
Bengio and others argue that networks should learn to activate only parts of themselves per example. The right idea, waiting for hardware and a killer host architecture.
Outrageously Large Neural Networks — Shazeer et al.
The modern blueprint: a sparsely-gated MoE layer with noisy top-k routing between LSTM layers, scaled to 137B parameters (unheard of in 2017) with only minor per-example compute cost. Also introduced the load-balancing auxiliary losses everyone still descends from.
GShard → Switch Transformer → GLaM
MoE meets the transformer at Google. GShard scales translation to 600B. Switch simplifies to k=1 routing and hits 1.6T parameters with a ~7× pre-training speedup over T5 at equal FLOPs. GLaM (1.2T, 97B active) beats GPT-3 on 1/3 of its training energy — the first proof MoE wins at frontier LLM scale.
ST-MoE & Expert Choice
The stabilization papers: router z-loss tames training blowups and fine-tuning collapse; Expert Choice inverts the routing question (experts pick tokens) to get perfect balance for free. MegaBlocks (late '22) makes token-dropping unnecessary with block-sparse kernels.
Mixtral 8×7B — MoE goes open
Mistral drops open weights that match Llama 2 70B and GPT-3.5 with 12.9B active parameters. Overnight, MoE stops being a Google-internal curiosity and becomes something anyone can download, dissect, and fine-tune.
The DeepSeek reformation
DeepSeekMoE (Jan) introduces fine-grained experts + shared experts. V2 (May) adds Multi-head Latent Attention and proves the economics: stronger than their dense 67B at 42.5% lower training cost. V3 (Dec) lands the haymaker: 671B/37B, FP8 training, auxiliary-loss-free load balancing, trained for ~$5.6M in GPU time. R1 builds reasoning on top a month later and makes the whole world notice. Meanwhile: Mixture-of-Depths routes compute across layers, and PEER scales to a million tiny experts.
The sparsity race
Everyone converts. Meta ships Llama 4 as MoE. Qwen3 introduces global-batch balancing. Kimi K2 opens a 1-trillion-parameter model (32B active). OpenAI's GPT-OSS returns to open weights as a 117B/5.1B MoE in 4-bit. LongCat-Flash makes compute per token dynamic with zero-computation experts. Qwen3-Next pushes activation below 4%. Mistral Large 3 (675B/41B) closes the year.
MoE is the default — and sparsity spreads to attention
Qwen3.5 (Feb, 397B/17B), Kimi K2.6 (Apr 20), DeepSeek-V4 (Apr 24: 1.6T/49B, hybrid compressed sparse attention, 1M context), MiniMax M3 (Jun 1), GLM-5.2 (Jun 13: 744B/40B) — and on July 16, Kimi K3: 2.8T parameters, 16 of 896 experts, the largest open-weight model ever shipped (full case study below). Every serious open-weight release this year is MoE — and each pairs sparse experts with sparse attention. §08 covers the frontier in detail.
The routing zoo
"Which experts see which tokens?" has three possible answers — tokens choose, experts choose, or nobody chooses — and a decade of variants inside each.
Routing is where nearly all MoE research concentrates, because it's where the two failure modes live: collapse (everything routes to a few experts — §06) and instability (discrete, shifting assignments make gradients noisy). Here is the field, compressed:
Token-choice top-k · the default
Each token independently picks its k highest-scoring experts. Simple, causal, composable — and inherently imbalance-prone, hence all of §06.
Noisy top-k · 2017
Add learnable Gaussian noise to scores before the top-k, so near-ties flip occasionally and losing experts still receive gradient. Exploration for routers.
Top-1 "Switch" routing · 2021
k=1: one expert per token. Halves routing compute and communication vs top-2; needs careful capacity factors and loss scaling but works shockingly well.
Expert-choice · 2022
Invert it: each expert selects its top-C tokens from the batch. Perfect load balance by construction, and important tokens can get more experts. Catch: selection looks across the whole batch/sequence — which peeks at future tokens, so it's awkward for causal decoding.
Hash / random routing · 2021
No learning at all: route by a fixed hash of the token ID. A humbling baseline — it captures a surprising share of MoE's benefit, proving that capacity itself does a lot of the work even before smart routing.
BASE layers · 2021
Treat routing as a global assignment problem and solve it with a linear program / auction each step: balance is exact and no tokens drop. Elegant; the dispatch cost and batch-coupling kept it niche.
Soft MoE · 2023
No discrete choice: each expert processes a learned weighted blend of all tokens. Fully differentiable, drop-free, stable — but every token influences every expert, which breaks causal masking, so it shines in vision, not LLM decoding.
ReLU / differentiable routing · 2024–26
Replace top-k + softmax with a ReLU gate: sparsity emerges from the activation itself, is fully differentiable, and the number of active experts can vary per token. Sparsity is steered with regularization. An active 2026 research thread, along with variational/Bayesian routers for calibrated expert selection.
Biased top-k (aux-loss-free) · 2024
Keep token-choice, but add a per-expert bias — used for selection only — that a feedback controller nudges to equalize load. Balance without a gradient penalty. The technique that defines the current era; full math in §07.
Zero-computation experts · 2025
Include identity "experts" that return the input untouched at zero FLOPs. The router can now decide a token is easy and spend nothing on it — compute per token becomes elastic. §08 has the details.
Load balancing — the problem that ate a decade
Left alone, routers collapse: a couple of experts win early, get more gradient, get better, win more. You end up paying for 256 experts and using 6. Most of MoE history is the fight against this feedback loop.
The collapse mechanism is a textbook rich-get-richer loop. Early in training, some expert is randomly slightly better for common tokens → the router sends it more tokens → it receives more gradient and improves faster → its router scores rise further. Uncorrected, load concentrates catastrophically. This is doubly fatal because experts are physically sharded across GPUs (§10): a hot expert means one device saturates while the rest of the cluster idles — you lose the quality of a big model and the efficiency of a sparse one.
Fix 1 — The auxiliary balance loss (2017 → standard for 7 years)
You can't differentiate "how many tokens went to expert i" (it's a count). The classic trick multiplies the non-differentiable count by its differentiable proxy — the router's mean probability — and penalizes their correlation:
Both \(f\) and \(P\) sum to one, so \(\sum f_i P_i\) is minimized exactly when both are uniform; gradient flows through \(P\), dragging the router toward spreading probability wherever load is lopsided. It works — and it hurts. The loss is a tax term added to the language-modeling objective: set \(\alpha\) high and you force the router to make genuinely worse expert choices for the sake of balance; set it low and you collapse. Seven years of MoE training was tuning that knob.
Fix 2 — Router z-loss (2022)
ST-MoE found that router logits love to explode into instability (giant logits → saturated softmax → brittle, jumpy routing, occasionally diverging runs, especially in bf16). The z-loss gently compresses the logsumexp of the logits:
Cheap, boring, and still in most 2026 training stacks in some form.
Fix 3 — Capacity factors and token dropping (2020) … and dropless kernels (2022)
Hardware wants fixed-size tensors, so GShard/Switch gave each expert a buffer of \(C = \phi \cdot kT/N_E\) token slots (\(\phi \approx\) 1.0–1.25). Tokens that overflow a popular expert are simply dropped — they skip the layer via the residual connection. Ugly but effective; at up-to-double-digit drop rates, surprisingly survivable. MegaBlocks later reformulated MoE as block-sparse matrix multiplication (grouped GEMM), letting each expert take a variable number of tokens with no dropping and no padding waste — the "dropless" style most modern open models train with. DeepSeek-V3 onward drops nothing.
Fix 4 — Change the batch, not the loss (2025)
Qwen3 asked a sharper question: balanced over what? Enforcing balance within every micro-batch forces every tiny slice of data to spread across all experts — which actively prevents experts from specializing by domain (a batch of pure code should hammer the code experts). Computing \(f_i\) over the global batch (synced across devices) relaxes this: specialization within a batch, balance in aggregate. Measurably better perplexity and much more interpretable experts.
A toy router over 8 experts with a rich-get-richer training signal baked in. Orange bars: share of tokens each expert actually receives (target line = uniform 12.5%). Teal strip: the router's raw preference — its learned specialization, before any correction.
Try all three: no balancing collapses; the aux loss balances load but flattens the teal preferences too (it distorts what the router learned); the bias method balances load while the teal skew survives.
The modern recipe — how DeepSeek rewired the field
Four ideas from one lab — fine-grained experts, shared experts, sigmoid gates, and bias-controlled balancing — became the template that nearly every 2025–26 frontier model now follows.
7.1 · Fine-grained experts: same FLOPs, exponentially more combinations
Mixtral-style MoE uses 8 big experts and picks 2. DeepSeekMoE's observation: slice each expert into \(m\) thinner ones and pick \(m\!\times\!k\) of \(m\!\times\!N_E\). Per-token FLOPs are identical — you run the same total width — but the router now composes knowledge from many small units instead of committing to two monoliths. The number of possible expert teams explodes:
| Configuration | Active width | Possible expert combinations |
|---|---|---|
| pick 2 of 8 (Mixtral-style) | 1× | 28 |
| pick 2 of 16 | 1× | 120 |
| pick 4 of 32 (½-width experts) | 1× | 35,960 |
| pick 8 of 64 (¼-width) | 1× | ≈ 4.4 billion |
The catch is systems-side: more, smaller experts means more dispatch bookkeeping and smaller matrix multiplies per expert — which is exactly what grouped-GEMM kernels and modern all-to-all stacks (§10) were built to absorb.
7.2 · Shared experts: stop making 64 copies of English grammar
If every expert must be able to handle commas, articles, and whitespace, you've duplicated the most common knowledge dozens of times — wasted capacity. DeepSeekMoE routes every token through one or two shared experts unconditionally, alongside its routed picks:
The shared path absorbs the universal patterns, freeing routed experts to differentiate. Adopters: DeepSeek V2/V3/V4, Kimi K2/K2.6, Llama 4 (its "1 shared + 1 routed" is the minimal version), Qwen3.6, MiniMax. Qwen3 notably dropped shared experts and compensated with global-batch balancing — proof the pieces are somewhat interchangeable, not gospel.
7.3 · The gate itself: softmax → sigmoid → √softplus
Softmax forces experts to compete: raising one score lowers all others, which gets awkward with 256+ experts where scores should be closer to independent "is this expert relevant?" judgments. DeepSeek-V3 switched the affinity to a sigmoid per expert, normalizing only across the selected top-k. DeepSeek-V4's report swaps the sigmoid for \(\sqrt{\mathrm{softplus}(\cdot)}\) — an unbounded, smooth positive affinity — a small change that reportedly trains better at trillion scale. The lesson: with fine-grained pools, the gate is an open design surface, not settled science.
7.4 · Auxiliary-loss-free balancing — the headline trick ★
Recall §06's dilemma: the balance loss keeps experts busy by corrupting the router's judgment. DeepSeek-V3's fix (with Wang et al. 2024) is almost insultingly simple. Split the router's two jobs apart:
The bias \(b_i\) is not a trained parameter and carries no gradient — it's a feedback controller (γ ≈ 10⁻³) that nudges the selection margin of overworked experts down and underworked ones up. Crucially, \(b\) only decides who gets picked; the mixture weight \(g_i\) still comes from the raw affinity \(s_i\), so the model's learned trust in each expert is never distorted, and no penalty term pollutes the language-modeling gradient. DeepSeek reported both better balance and better loss than aux-loss training — the rare free lunch. You watched exactly this in the §06 lab: load flattens, the teal preferences survive.
Two footnotes from the V3/V4 reports: a tiny sequence-wise balance loss (α ≈ 10⁻⁴) still rides along, purely to prevent pathological imbalance within a single sequence; and V3's node-limited routing (each token's experts confined to ≤4 server nodes, to cap network traffic) was removed in V4 after a parallelism redesign — the systems finally caught up with the algorithm. By 2026 there's even formal convergence analysis of the bias controller; the trick graduated from hack to theory.
The 2026 default MoE layer, assembled
256–512 fine-grained routed experts + 1 shared expert · top-6 to top-10 selection · sigmoid-family affinity normalized over the selected set · bias-controlled aux-loss-free balancing + a whisper of sequence-wise loss · dropless dispatch over a global-batch view · roughly 3–6% of parameters active per token. If you sketch a new MoE model today, this is the starting point you deviate from.
The frontier: what's new in 2025–26
Five live fronts: sparsity as a scaling law, elastic per-token compute, million-expert lookups, the MoE-ification of attention — and the flagship that combines them all.
8.1 · Sparsity became a scaling law
Define sparsity as total ÷ active parameters. The empirical finding driving 2025–26 design — made explicit in Kimi K2's report and formalized by 2026 "holistic MoE scaling law" work — is that at fixed active FLOPs, increasing sparsity keeps lowering loss, with diminishing returns and rising systems cost setting the practical knee. So labs pushed hard:
- Mixtral 8×7B (2023): 3.6× — a quarter of the model fires per token.
- DeepSeek-V3 (2024): 18× — top-8 of 256 experts (48× at the expert level).
- Kimi K2 (2025): 32× — 1.04T total, 32.6B active; chose expert-sparsity 48 (384/8) from their own sparsity scaling curves.
- Qwen3-Next-80B-A3B (2025): 27× — 512 experts, top-10 + shared, ~3.7% activation; beats dense Qwen3-32B at under a tenth of its training cost.
- 2026 flagships: DeepSeek-V4-Pro ~33×, Kimi K2.6 ~31×, MiniMax M3 ~23×, GLM-5.2 ~19× — and on July 16, Kimi K3 blew past them all at ≈56× (16 of 896 experts; full case study below). Even small models converted: Qwen3.6-35B-A3B runs ~120 tok/s on a single RTX 4090, and Google's Gemma line went MoE.
8.2 · Elastic compute: not every token deserves the same budget
Classic MoE spends identical FLOPs on "the" and on the keystone token of a proof. Two techniques break that:
Zero-computation experts · LongCat-Flash (2025)
Mix real FFN experts with identity experts that cost nothing. The router's top-k can land on zeros for easy tokens, so active parameters float per token — 18.6B to 31.3B on a 560B model, averaging ~27B — with a PID-style controller on the expert biases holding the mean compute on budget. Difficulty-aware spending, learned end to end.
Mixture-of-Depths · Google (2024)
Route around entire layers: at each block, a capacity-limited router decides which tokens get processed and which ride the residual stream through untouched. Combine with expert routing (MoDE) and the model allocates depth and width simultaneously. Influential idea; harder to serve, so adoption trails MoE proper.
8.3 · The million-expert direction
If finer granularity keeps winning (§7.1), the limit case is intriguing: PEER (DeepMind, 2024) routes over ~a million single-neuron experts using product-key retrieval — the top-k search factorizes into two cheap searches over √N keys, so selection stays sublinear. Meta's memory layers reach the same destination from another road: replace some FFNs with a giant trainable key-value lookup (hundreds of billions of "memory" parameters touched only sparsely). Both blur the line between computing an answer and looking it up — expect this thread to matter as models chase parametric knowledge without compute.
8.4 · Sparsity ate attention too
The FFN was ~⅔ of the FLOPs problem; long context made attention the other villain. 2025–26's answer reuses the MoE playbook — score chunks, keep the top-k, skip the rest:
- MoBA (Moonshot, 2025): each query attends to its top-k blocks of KV cache — literally "mixture of block attention."
- DSA (DeepSeek-V3.2, 2025): a lightning indexer picks which tokens deserve full attention; near-linear long-context cost.
- CSA + HCA (DeepSeek-V4, 2026): a two-tier compressed hierarchy over the context — at 1M tokens, 27% of the inference FLOPs and 10% of the KV cache of V3.2.
- IndexShare (GLM-5.2, 2026): shares the sparse-attention indexer's work across layers, cutting per-token compute ~2.9× at 1M context. MSA (MiniMax M3): >9× prefill and >15× decode speedups at 1M vs. its predecessor.
- KDA (Moonshot, 2025 → K3 2026): the linear-attention road — Kimi Delta Attention layers interleaved with periodic gated full-attention (MLA) layers, credited by Moonshot with up to 6.3× faster decoding at 1M-token context in K3.
The deep pattern of the decade: learned top-k selection over interchangeable units — experts, layers, context blocks — is how transformers stopped paying dense prices for everything.
8.5 · The training stack caught up
Trillion-parameter MoE also needed boring-but-critical infrastructure wins: the Muon optimizer family (K2's MuonClip pretrained 15.5T tokens with zero loss spikes; V4 adopts Muon; K3 extends it to Per-Head Muon, optimizing attention heads independently), low-precision expert weights as a first-class design axis (V3 pioneered FP8 training; V4 does FP4 quantization-aware training on experts; K2.6 ships native INT4; GPT-OSS's MXFP4 squeezes 117B parameters onto a single 80 GB GPU; K3 trains quantization-aware from the SFT stage with MXFP4 weights + MXFP8 activations), stability scaffolding like V4's manifold-constrained hyper-connections (mHC), and multi-token prediction heads almost everywhere. None of these are "MoE techniques," but they're why 30× sparsity is deployable rather than a paper result.
Mini case study — DeepSeek-V4-Pro (Apr 24, 2026): the spring flagship, dissected
The most complete expression of everything above, in one open-weights (MIT) release:
- Scale: 1.6T total / 49B active (~3%); Flash sibling at 284B/13B; ~33T training tokens; native 1M context.
- MoE core: DeepSeekMoE fine-grained + shared experts (third-party breakdowns report ~top-6 of 385); affinity function upgraded from sigmoid to √softplus; aux-loss-free bias balancing + light sequence-wise loss; node-routing limits removed entirely.
- Attention: hybrid CSA + HCA sparse attention — 27% FLOPs, 10% KV cache vs. V3.2 at 1M tokens.
- Stack: Muon optimizer, mHC residual topology, FP4-aware expert weights.
- Result: frontier agentic-coding scores (80.6% SWE-bench Verified reported) at $0.435 per million input tokens — priced like a mid-size dense model, because per-token it is one.
Config details marked "reported" come from third-party writeups of the preview release; check the tech report (§12) before load-bearing use.
Kimi K3, end to end: a concrete MoE teardown
Every technique in this guide, running in one shipping system. Moonshot's Kimi K3 is the largest open-weight model ever released — here it is as a worked example: the review, the routing math, the new balancing trick, and the serving economics.
★.1 · The review: what shipped, and where it lands
On July 16, 2026, Moonshot AI released Kimi K3: a 2.8-trillion-parameter Mixture of Experts with native vision, a 1-million-token context window, and always-on reasoning (launched at max thinking effort). It is the first open model in the 3T class — Kimi models have set the open-model size ceiling for nine of the past twelve months. Moonshot is unusually candid about position: overall, K3 trails Claude Fable 5 and GPT-5.6 Sol, while consistently beating everything else in their 35-benchmark suite. Its distinctive strength is long-horizon autonomous work: in Moonshot's demos it optimized GPU kernels competitively with Fable 5 over 24-hour sandboxed runs, built a working Triton-like GPU compiler ("MiniTriton") from scratch, and designed and verified a small inference chip in a single 48-hour autonomous session.
Early independent signals cut both ways: K3 debuted at #1 on LMArena's Frontend Code Arena and leads benchmarks like BrowseComp (90.4 with the full 1M window), but AA-Omniscience testing found accuracy jumping 33% → 46% over K2.6 while the hallucination rate rose 39% → 51% — better at answering, worse at knowing when not to. Moonshot's own listed limitations: sensitivity to preserved thinking history, "excessive proactiveness" on ambiguous tasks, and a user-experience gap versus Fable 5 and Sol. Pricing is $0.30 / $3.00 / $15.00 per million tokens (cache-hit input / cache-miss input / output) — premium by Chinese-lab standards, a fraction of frontier US rates. Full weights are due July 27; the technical report is still pending, so several figures below are launch-blog and third-party level, flagged where so.
★.2 · The mixture, by the numbers
| Dimension | Kimi K2 / K2.6 (2025–26) | Kimi K3 (Jul 2026) |
|---|---|---|
| Total parameters | 1.04T | 2.8T (≈2.7×) |
| Active per token | 32.6B | ~50B (third-party reported) |
| Routed experts | 384 (+1 shared) | 896, plus an always-on shared path per the architecture diagram |
| Experts per token (k) | 8 | 16 |
| Expert activation ratio | 2.1% | 1.8% |
| Sparsity (total ÷ active) | 32× | ≈56× |
| MoE framework | DeepSeek-V3-style, bias balancing | Stable LatentMoE + Quantile Balancing |
| Attention | MLA (full) | KDA hybrid linear, interleaved ~3:1 with gated MLA layers (per diagram) |
| Optimizer | MuonClip | Per-Head Muon |
| Precision | native INT4 (K2.6) | MXFP4 weights + MXFP8 activations, QAT from SFT onward |
| Context / modality | 256K · text | 1M · native vision |
★.3 · Follow one token through a K3 block
Instantiate §03's general machinery with K3's numbers. A token representation \(x\) first passes attention — usually a Kimi Delta Attention layer (linear-time, holding long context in a compact recurrent state), periodically a gated full-attention MLA layer to restore exact recall. Then the MoE FFN: the router scores all 896 routed experts, the balancing scheme sets the cut, sixteen experts run, 880 stay cold, and the shared path fires unconditionally:
The routing space this buys is worth pausing on. §7.1's combinatorics table topped out at "pick 8 of 64 ≈ 4.4 billion teams." K3's pool gives, per layer, per token:
— about 10¹⁹ times the routed diversity of DeepSeek-V3's 8-of-256, at a lower activation ratio. This is the fine-grained-experts thesis (§7.1) pushed further than anyone has shipped it: same order of per-token compute as a ~50B dense model, with the router composing that compute from an astronomically larger menu.
★.4 · The problem it solves, priced out
Run §02's cost model on K3. A dense 2.8T transformer would burn roughly \(2N \approx 5.6\times10^{12}\) FLOPs per token at inference; K3's ~50B active parameters cost \(\approx 1.0\times10^{11}\) — a 56× compute discount on every token, forever, which is the only reason a 2.8T open model is economically thinkable at all. Moonshot additionally claims the whole architectural package (KDA + Attention Residuals + the sparsity jump + recipe changes) converts compute into capability about 2.5× more efficiently than K2 — a scaling-efficiency claim, on top of the sparsity discount.
The other §02 currency — memory — is where K3 bites. At MXFP4 (≈4.25 bits/param with block scales), 2.8T parameters is roughly 1.5 TB of weights: the VRAM tax of §11 at its historical maximum. Moonshot's serving guidance follows directly: deploy on supernodes of 64+ accelerators (~23 GB of expert weights per device before KV cache and activations), keep the all-to-all inside a big high-bandwidth domain (§10), and lean on batch appetite — at batch 1 a decode step reads only the ~26 GB of active weights, and with many concurrent users nearly all 896 experts earn their keep each step. Two more systems tricks close the loop: Mooncake-style disaggregated prefill/decode with KDA-aware prefix caching (contributed to vLLM) yields >90% cache-hit rates on coding traffic — which is exactly why cache-hit input is priced 10× below cache-miss.
★.5 · What's genuinely new: Quantile Balancing
K3 also moves the §06 story forward a step. The lineage so far: the aux loss (2017) bought balance by taxing quality; the bias controller (2024) removed the tax but introduced a heuristic update with a sensitive step size γ. K3's Quantile Balancing "derives expert allocation directly from router-score quantiles, eliminating heuristic updates and a sensitive balancing hyperparameter" (Moonshot's description — full math awaits the tech report). Read against the history, the direction is clear: instead of penalizing imbalance or correcting it after the fact, set the selection threshold from the score distribution itself so balance falls out of how the cut is made. Moonshot pairs it with a fully balanced expert-parallel training scheme — static tensor shapes, no host synchronization on the critical path — treating balance as a systems contract, not just a training nicety. That combination is what "stable and efficient training at 2.8T scale" cashes out to.
★.6 · Scorecard, caveats, and what it proves
Trust levels, explicitly: from Moonshot's launch blog — 2.8T total, 16-of-896 Stable LatentMoE, shared + routed experts, KDA + AttnRes, Quantile Balancing, Per-Head Muon, MXFP4/MXFP8 QAT, 1M context, ~2.5× scaling efficiency vs K2, pricing, July 27 weights. Third-party reported — ~50B active (and hence the 56× figure), the ~3:1 KDA:MLA interleave, Modified-MIT licensing. The trade-offs echo §11 almost line by line: capability and hallucination rose together; fine-tuning a 2.8T MoE is out of reach for nearly everyone (expect the ecosystem to run on distilled students within weeks of the weight drop); and the VRAM tax means "open weights" here means open to organizations with a supernode, plus whatever the quantization community manages next.
Scoreboard: the real models, by the numbers
One chart tells the story of the last three years: sparsity — total parameters divided by active — climbed from ~4× to ~56× while quality went up, not down.
| Model | Date | Total | Active | Sparsity | Experts (routed+shared) | k | Balancing | Notable |
|---|---|---|---|---|---|---|---|---|
| Switch-C | 2021 | 1.57T | (top-1) | — | 2048 | 1 | aux + capacity | 7× T5 pretrain speedup |
| GLaM | 2021 | 1.2T | 97B | 12× | 64 | 2 | aux loss | ⅓ of GPT-3's training energy |
| Mixtral 8×7B | 2023 | 46.7B | 12.9B | 3.6× | 8 | 2 | aux loss | open-weights breakout |
| DeepSeekMoE-16B | 2024 | 16.4B | 2.8B | 5.9× | 64 + 2 | 6 | aux loss | fine-grained + shared debut |
| DeepSeek-V2 | 2024 | 236B | 21B | 11× | 160 + 2 | 6 | aux, device-limited | MLA debut; −42.5% train cost vs dense 67B |
| DeepSeek-V3 / R1 | 2024 | 671B | 37B | 18× | 256 + 1 | 8 | bias (aux-free) | FP8; ~$5.6M GPU bill |
| Llama 4 Maverick | 2025 | 400B | 17B | 24× | 128 + 1 | 1 | aux loss | alternating dense/MoE; multimodal |
| Qwen3-235B-A22B | 2025 | 235B | 22B | 11× | 128 + 0 | 8 | global-batch aux | dropped shared experts |
| Kimi K2 | 2025 | 1.04T | 32.6B | 32× | 384 + 1 | 8 | bias | MuonClip; 15.5T tokens, no spikes |
| GPT-OSS-120B | 2025 | 117B | 5.1B | 23× | 128 | 4 | aux loss | MXFP4 → runs on one 80 GB GPU |
| LongCat-Flash | 2025 | 560B | 18.6–31.3B | ~21× | FFN + zero-compute | dyn | PID bias | elastic per-token compute |
| Qwen3-Next-80B | 2025 | 80B | 3B | 27× | 512 + 1 | 10 | global-batch | hybrid linear attention; <1/10 train cost of dense 32B |
| Mistral Large 3 | 2025 | 675B | 41B | 16× | — | — | — | serves on one 8-GPU node |
| Qwen3.5-397B | 2026 | 397B | 17B | 23× | 512 | — | — | multimodal flagship |
| Kimi K2.6 | 2026 | 1T | 32B | 31× | 384 + 1 | 8 | bias | native INT4; agent swarms |
| DeepSeek-V4-Pro | 2026 | 1.6T | 49B | 33× | ~385* | ~6* | bias, √softplus | CSA+HCA sparse attention; 1M ctx |
| MiniMax M3 | 2026 | 230B | 9.8B | 23× | 256 | — | — | MSA; native multimodal |
| GLM-5.2 | 2026 | 744B | 40B | 19× | 256 | 8 | — | top open scorer until K3; IndexShare |
| Kimi K3 | 2026 | 2.8T | ~50B* | ~56× | 896 + shared | 16 | quantile | largest open model ever; KDA; 1M ctx; MXFP4 — see ★ |
* third-party reported (V4 preview config; K3 active-parameter count); "—" = not disclosed at time of writing. Closed models went MoE first and quietly: Google's reports confirm Gemini 1.5+ as sparse MoE, and GPT-4 has long been reported (never confirmed) as a ~1.8T mixture of 16 experts.
Systems reality: where MoE is actually won or lost
On paper, MoE is one equation. In a datacenter, it's a distributed-systems problem: experts live on different GPUs, and every layer is a network round-trip.
Expert parallelism and the all-to-all tax
No single GPU holds 1.6T parameters, so experts are sharded across devices (expert parallelism). Every MoE layer then runs: route → all-to-all dispatch (each GPU mails its tokens to whichever GPUs host their chosen experts) → expert compute → all-to-all combine (results mailed back). That's two collective network operations per layer per step. The modern stack exists to hide this: custom dispatch kernels (DeepEP-style), pipeline schedules that overlap communication with computation (DeepSeek's DualPipe), shortcut topologies that let the network transfer run under an adjacent dense block (LongCat's ScMoE), and — until systems matured — routing constraints like V3's 4-node limit, now retired in V4.
Inference: load balancing never ends
Training balance (§06–07) doesn't guarantee serving balance — real traffic is bursty and domain-skewed. Production stacks re-balance live: EPLB-style placement replicates the hottest experts on extra GPUs and reshuffles expert-to-device assignment; prefill and decode are often disaggregated onto separate pools tuned for their different arithmetic profiles. MoE also has a strong batch appetite: with many concurrent requests, every expert gets work each step and utilization soars — one reason MoE economics favor high-traffic API serving.
The memory ledger, with real numbers
- Hold total, read active. Mixtral 8×7B in fp16: ~94 GB to hold, ~26 GB of weights read per generated token. Decode is bandwidth-bound, so tokens flow at small-model speed while capacity stays big-model.
- Quantization is the great equalizer. 4-bit expert weights changed who can run these: GPT-OSS-120B (MXFP4) on a single 80 GB GPU; Kimi K2.6 shipping native INT4; Qwen3-Coder-480B locally in ~250 GB; Qwen3.6-35B-A3B at ~120 tok/s on one RTX 4090. In 2026, trillion-class capability rents by the node, and 35B-class MoE runs on a gaming card.
- KV cache is the other memory bill — which is why MoE labs also invented MLA and the sparse-attention family (§8.4): sparse FFN and lean attention are one co-designed budget.
Trade-offs, failure modes, and comfortable myths
MoE won because its costs moved somewhere payable — not because they vanished. Know where they went.
The VRAM tax is real
You pay memory for parameters you mostly don't use. If your deployment is a single small GPU with tight RAM and one user, a good dense model can still be the right call — MoE's economics assume the full expert set fits somewhere cheap to keep.
Fine-tuning is touchier
Small, narrow SFT datasets can re-collapse routing (a few experts absorb the whole task) and MoEs historically overfit faster than dense peers (documented back in ST-MoE). Common practice: freeze or gently regularize the router, watch per-expert load during tuning, and prefer larger/mixed tuning data. RL on MoE is its own emerging subfield.
Communication can eat the win
All-to-all across slow interconnects, tiny batches, or naive expert placement can burn every FLOP saved. MoE performance is an engineering outcome, not an architecture guarantee — the same checkpoint can be fast or sluggish depending entirely on the serving stack.
Batch-coupling quirks
Capacity-limited and expert-choice schemes make one sequence's routing depend on its batchmates — a reproducibility and eval subtlety. Dropless, per-token routing (the modern default) mostly retired this, but it lurks in older stacks.
Myth check: "the biology expert"
Interpretability work on Mixtral found little clean topical specialization: experts specialize by token identity, position, and syntax role far more than by subject matter, and consecutive tokens often share experts. Human-legible domain experts are the exception, not the rule — though global-batch balancing (Qwen3) measurably increases domain-level specialization, and 2026's "Super Experts" work shows importance is concentrated in a small subset whose removal cripples the model (a gift for compression, a warning for pruning). Treat the router as a learned statistical hash, not an org chart.
Cheat sheet & primary sources
Thirty-second glossary
- expert
- one FFN among many in a layer; runs only when routed to.
- router / gate g(x)
- tiny linear scorer choosing top-k experts per token, per layer, and weighting their outputs.
- active vs total params
- compute cost per token vs stored capacity; sparsity = total ÷ active (2026 frontier: ~20–33×).
- shared expert
- always-on expert absorbing universal patterns so routed experts can specialize.
- fine-grained experts
- many thin experts instead of few wide ones — combinatorially richer routing at equal FLOPs.
- aux balance loss / z-loss
- classic penalties keeping load spread and logits sane — at a quality tax.
- aux-loss-free (bias) balancing
- feedback-controlled per-expert bias applied to selection only; the current standard.
- capacity factor / dropless
- old fixed token buffers with overflow drops vs modern variable-size grouped-GEMM dispatch.
- expert parallelism / all-to-all
- experts sharded across GPUs; two network collectives per MoE layer.
- zero-computation expert
- identity expert costing nothing — makes per-token compute elastic.
Primary sources (chronological)
Where this is going
The through-line of thirty-five years is one sentence: stop paying dense prices for sparse needs. First it was the FFN; now it's attention, depth, and precision. The open questions for the next cycle are already visible — how far the sparsity scaling law bends before it breaks, whether million-expert lookup structures merge computation with retrieval, whether routers can be made calibrated and auditable, and whether per-token elastic compute becomes the default rather than the exception. Whatever the answers, they'll be reviewed here in the same spirit: with the math shown and the receipts attached.