The model, one layer at a time
Primer page 3. What Muse Spark Open actually computes, just deeply enough to see why each piece costs what it costs.
A stack of 62 layers
Inside the model, every token is a list of 8,192 numbers: its state. The model is a stack of 62 layers, and each layer nudges the state a little. After the last layer, the state is turned into a choice of next token. Every layer does the same two things in the same order: look back at earlier tokens, then think about what it found.
| Name on the plan pages | What it is | Stored as | Plan page |
|---|---|---|---|
| qkv_gate, o, pooler, unpooler | Four fixed matrices. Every token is multiplied by all of them. | 16-bit numbers | bf16_proj |
| router | A small matrix that scores all 256 experts for a token. | 32-bit numbers | bf16_proj |
| attention | Reads the notes on earlier tokens. | 8-bit notes | attention_decode |
| experts (MoE) | 256 small networks; each token uses 8. | 4-bit numbers | moe_w4a16 |
| the whole layer | All of the above chained together. | decode_layer |
Looking back: attention and its notes
Think of a library. Every earlier token left behind an index card (its key) and a note (its value). The new token writes a question. Attention compares the question with every index card, decides which cards match best, and blends the notes behind those cards into the new token's state.
The pile of cards and notes is the KV cache. It exists so that earlier tokens are never recomputed. The price is that every new token has to read the pile: 256 bytes per earlier token, per layer, on each GPU.
Not to scale: in an 80,000-token conversation the blue part would be a sliver.
- Sliding layers (46 of the 62) look only at the most recent 2,048 tokens. That is half a megabyte: almost free. Their cost is mostly fixed overhead.
- Full layers (16 of the 62) look at everything. For an 80,000-token conversation that is about 20 MB per layer per user. With 32 users it is roughly 600 MB per layer, more than any other part of the layer reads.
- Eight questions per pile. On each GPU, 16 "question heads" share 2 piles of cards, so one pass over a pile can answer eight questions. The plan pages call this grouped-query attention, and a good kernel reads each pile once for all eight.
Thinking: 256 experts, 8 at a time
The second half of a layer is where most of the model's knowledge lives. Instead of one enormous network, each layer has 256 smaller ones called experts. A router scores them for each token and picks the best 8. Only those 8 run.
That is how a model with 816 billion numbers can answer using only about 42 billion per token. It has two consequences for speed:
- Nobody knows which expert weights will be needed until the router has spoken. The fetch cannot start early.
- If several tokens pick the same expert, its weights are fetched once and used for all of them. More tokens means more sharing.
On stock SGLang the experts are about half of a layer's time at one user, and more at higher loads. That is why they are first on the plan.
Eight GPUs, one model
The model is far too big for one GPU, so eight share it in two ways at once:
- Slices (the plan's "TP8"). Each GPU holds one eighth of every big matrix and does its eighth of the multiplication. The eight partial results are then combined.
- Experts (the plan's "EP8"). Each GPU holds 32 of the 256 experts. A token's 8 experts may be spread over several GPUs.
After each part of a layer the GPUs exchange results, and nobody can move on until the slowest has finished. If the router happens to send most tokens to experts on one GPU, that GPU sets the pace for all eight. The contest's kernel judge tested a single GPU and always made it the busy one, with the exchange left out. A real server has to pay for the exchange too.
Reading and writing
- Prefill is reading the incoming message: hundreds or thousands of tokens pushed through all 62 layers in one batch. With that many tokens there is plenty of arithmetic per fetched byte. It decides how long the user waits before the reply starts.
- Decode is writing the reply, a few tokens per step. Here the fetch dominates, as page 2 showed. It decides how fast the reply streams.
Draft and check
Fetching a layer's weights costs the same whether one token or eight are waiting for them. So the server uses a trick called speculative decoding. A small, fast draft model guesses the next 8 tokens. The big model then checks all 8 positions in a single pass, and keeps the guesses up to the first one it disagrees with.
- The answer does not change. Whatever comes out is what the big model would have written on its own. The draft only saves time when it guesses right.
- It is cheap. On stock SGLang a layer takes about 171 µs for one token and about 267 µs for eight. That is 1.6 times the cost for up to eight times the tokens.
- It shapes everything else. With the draft on, every decode step is a batch of 8 tokens per user. So 1, 16 and 32 users mean 8, 128 and 256 tokens per step. Those are the sizes the plan pages keep quoting, under the name "verify".
What one step costs
| Users | Tokens checked per step | GPU time per step, stock |
|---|---|---|
| 1 | 8 | about 16.6 ms |
| 16 | 128 | about 35.2 ms |
| 32 | 256 | about 51.1 ms |
Thirty-two times the users costs only three times the time, because the weight fetch is shared. That gap is the reason servers batch users together, and the trade it creates between speed for each user and total output is the subject of page 6.
- A layer looks back (attention) and then thinks (8 of 256 experts), with four fixed matrices in between.
- Sliding layers are cheap; full layers get expensive with long conversations and many users.
- Experts are the largest cost, and which ones are needed is known only at the last moment.
- Eight GPUs must wait for the slowest at every exchange.
- Draft-and-check turns every step into an 8-token batch per user.
Model facts are from the task descriptions and the SGLang team's posts, as collected on the overview; the per-layer and per-step times are from the decode-layer and serving pages. For more depth, see Inside the transformer decoder and the mixture-of-experts field guide on this site. Next: number formats.