The model, one layer at a time

Primer page 3. What Muse Spark Open actually computes, just deeply enough to see why each piece costs what it costs.

A stack of 62 layers

Inside the model, every token is a list of 8,192 numbers: its state. The model is a stack of 62 layers, and each layer nudges the state a little. After the last layer, the state is turned into a choice of next token. Every layer does the same two things in the same order: look back at earlier tokens, then think about what it found.

A token's state: 8,192 numbers qkv_gate: form a question, a key and a value Attention: compare with earlier tokens o: fold what was found back into the state Router: choose 8 of the 256 experts Pooler: shrink 8,192 numbers to 4,096 8 experts: small networks, 4-bit weights Unpooler: grow back to 8,192 Updated state, passed to the next layer look back think KV cachenotes on everyearlier token
Name on the plan pagesWhat it isStored asPlan page
qkv_gate, o, pooler, unpoolerFour fixed matrices. Every token is multiplied by all of them.16-bit numbersbf16_proj
routerA small matrix that scores all 256 experts for a token.32-bit numbersbf16_proj
attentionReads the notes on earlier tokens.8-bit notesattention_decode
experts (MoE)256 small networks; each token uses 8.4-bit numbersmoe_w4a16
the whole layerAll of the above chained together.decode_layer

Looking back: attention and its notes

Think of a library. Every earlier token left behind an index card (its key) and a note (its value). The new token writes a question. Attention compares the question with every index card, decides which cards match best, and blends the notes behind those cards into the new token's state.

The pile of cards and notes is the KV cache. It exists so that earlier tokens are never recomputed. The price is that every new token has to read the pile: 256 bytes per earlier token, per layer, on each GPU.

16 full layers read all of it 46 sliding layers read only the last 2,048 tokens the conversation so far

Not to scale: in an 80,000-token conversation the blue part would be a sliver.

Thinking: 256 experts, 8 at a time

The second half of a layer is where most of the model's knowledge lives. Instead of one enormous network, each layer has 256 smaller ones called experts. A router scores them for each token and picks the best 8. Only those 8 run.

That is how a model with 816 billion numbers can answer using only about 42 billion per token. It has two consequences for speed:

On stock SGLang the experts are about half of a layer's time at one user, and more at higher loads. That is why they are first on the plan.

Eight GPUs, one model

GPU 1slices +32 experts GPU 2slices +32 experts GPU 3slices +32 experts GPU 4slices +32 experts GPU 5slices +32 experts GPU 6slices +32 experts GPU 7slices +32 experts GPU 8slices +32 experts after each part: swap results, then wait for the slowest GPU

The model is far too big for one GPU, so eight share it in two ways at once:

After each part of a layer the GPUs exchange results, and nobody can move on until the slowest has finished. If the router happens to send most tokens to experts on one GPU, that GPU sets the pace for all eight. The contest's kernel judge tested a single GPU and always made it the busy one, with the exchange left out. A real server has to pay for the exchange too.

Reading and writing

Draft and check

Fetching a layer's weights costs the same whether one token or eight are waiting for them. So the server uses a trick called speculative decoding. A small, fast draft model guesses the next 8 tokens. The big model then checks all 8 positions in a single pass, and keeps the guesses up to the first one it disagrees with.

Draft guessesBig model says the quick brown fox jumps past a lazy ✓ same ✓ same ✓ same ✓ same ✓ same ✗ over dropped dropped 5 guesses kept + 1 corrected = 6 tokens from one pass of the big model

What one step costs

UsersTokens checked per stepGPU time per step, stock
18about 16.6 ms
16128about 35.2 ms
32256about 51.1 ms

Thirty-two times the users costs only three times the time, because the weight fetch is shared. That gap is the reason servers batch users together, and the trade it creates between speed for each user and total output is the subject of page 6.

What to take away.

Model facts are from the task descriptions and the SGLang team's posts, as collected on the overview; the per-layer and per-step times are from the decode-layer and serving pages. For more depth, see Inside the transformer decoder and the mixture-of-experts field guide on this site. Next: number formats.