Start here: the challenge in plain words
A primer for readers who have never written GPU code. Six short pages, one idea at a time, with pictures and things to drag. Nothing harder than division. This first page stands on its own: read only this one and you will still have the whole picture.
- say in two sentences why generating text is slow, and why a single user is the hard case;
- work out the shortest time physics allows for a job from its size in bytes;
- sort any trick on the plan pages into one of three kinds, and spot the ones the rules forbid;
- estimate what speeding up one part does to the whole;
- read labels such as
v32sand words such as "floor" on the plan pages.
On this page The short version Follow one reply Why it is slow Three ways to go faster Which part first Rules and scores Reading the plan pages Try it yourself Check yourself Study path Glossary
The short version
A language model writes one token (roughly a piece of a word) at a time. To produce each token it has to run its whole network, and a network this size does not fit in any fast memory. Its numbers must be hauled in from the GPU's main memory again for every token. Most of the time the expensive calculators inside the GPU sit idle, waiting for numbers to arrive.
The challenge is to make that faster for one model (Meta's Muse Spark Open) on one machine (eight NVIDIA H200 GPUs), without changing a single answer. First by rewriting the small GPU programs that do the work, called kernels. Then by improving the server that feeds them, an open-source project called SGLang.
| The cast | What it is | Numbers worth remembering |
|---|---|---|
| The model | Muse Spark Open, a language model whose learned numbers (its weights) are public. | 816 billion weights, about 42 billion used for any one token, 62 layers. |
| The machine | One server with eight H200s. A GPU is a chip built to do huge amounts of simple arithmetic at once. | Per GPU: 141 GB of memory, 132 workers, about 4.3 MB delivered per microsecond. |
| The server | SGLang, the program that takes requests and runs the model. Unmodified SGLang is called stock. | Stock is the baseline. Every "2.4×" on this site means 2.4 times faster than stock. |
| The contest | Two rounds. Kernels: speed up one layer's programs on one GPU. Serving: speed up the whole server on all eight. | The kernel round closed on 4 November 2026. The serving round runs from 12 to 19 November. |
Follow one reply from start to finish
Prefill happens once per message. The step repeats dozens of times per reply, so the step is where the time goes.
Three words in the picture need a line each.
- Prefill. Before writing anything, the model reads the conversation once and keeps notes on every token, so it never has to re-read it.
- Layer. The model is a stack of 62 layers. Each one looks back at earlier tokens and nudges the current token's numbers a little. One trip through all 62 is one pass.
- Draft and check. A small, fast model guesses the next 8 tokens. The big model checks all 8 in a single pass and keeps the guesses up to its first disagreement, plus its own correction. The reply is exactly what the big model would have written alone. The draft only saves passes.
Worked example: a 220-token reply
- Suppose 5 tokens survive each check. That figure is assumed for illustration; the real one changes with the text.
- 220 tokens ÷ 5 per step = 44 steps.
- On stock SGLang one step costs about 16.6 ms of GPU work (ms is a thousandth of a second). 44 × 16.6 ms = 730 ms.
- 220 tokens in 0.73 seconds is about 300 tokens per second.
Now change the numbers yourself. The three buttons are the three step times this page keeps coming back to.
This counts the big model's GPU work only, for one user. A real server adds the draft model, the talk between the eight GPUs and its own bookkeeping, so real speeds are lower. The 220-token length is the one participants report for the serving round.
Why it is slow
Picture a kitchen with 132 very fast cooks. The ingredients are in a warehouse down the road, and there is one road. However fast the cooks are, dinner cannot be ready before the trucks arrive.
The cooks are the GPU's 132 workers. The ingredients are the weights. The warehouse is the GPU's main memory, the only place big enough to hold them. The road is memory bandwidth: how many bytes can leave main memory each second. On an H200 the road carries about 4.3 MB every microsecond (µs, a millionth of a second).
shortest possible time = bytes to fetch ÷ road speed
That one division is the most used sum on this site. Its answer is called the floor. No code can beat the floor, however clever, unless it fetches fewer bytes.
Worked example 1: one matrix
Each layer has a table of weights called qkv_gate, with 2,304 rows and 8,192 columns. Each weight is a 16-bit number, which is 2 bytes.
- Size: 2,304 × 8,192 × 2 bytes = 37.7 million bytes = 37.7 MB.
- Floor: 37.7 MB ÷ 4.3 MB per µs = 8.8 µs.
- The arithmetic on those weights, for one token, takes the GPU's calculators about 0.04 µs at their rated speed.
Fetching takes over 200 times longer than calculating. That is what "the calculators sit idle" means, and it is why a faster calculator would not help here.
Worked example 2: one layer, then one step
Here is everything one GPU must fetch to push one token through one layer, with no draft model. Change the road speed, pack the weights tighter or lengthen the conversation, and watch the floor move.
| What is fetched | MB | µs | Share of the layer |
|---|
Three things to try.
- Untick the box. The layer grows by 32.5 MB and 7.6 µs. Over 62 layers that is nearly half a millisecond per step, and ticking it again wins that back without changing any result. This is the first of the three ways to go faster.
- Drag the conversation from 2,000 to 200,000 tokens. The step grows by under 0.2 ms. For one user, looking back is cheap. It gets expensive only when many users each bring a long history.
- Slide the road speed. Everything scales with it. Bytes and road speed are the only two things in the formula.
Sizes are rows × columns × bytes per number, from the shapes on the decode-layer page; the expert figure is that page's estimate. With the box unticked, the road at its rated 4.8 MB per µs and an 80,000-token conversation, the table lands within a few percent of the organizers' own floor for this case: 40.0 µs for a sliding layer and 2.5 ms for a step.
The whole contest in three numbers
With draft and check, a step carries 8 tokens, not 1. The big tables are fetched once however many tokens there are, but 8 tokens choose more experts between them, so more bytes are needed. The organizers put the floor for that step at 62.3 µs for each of the 46 sliding layers and 66.4 µs for each of the 16 full ones.
| What | Time per step | Where the number comes from |
|---|---|---|
| Physical minimum | about 3.9 ms | 46 × 62.3 µs + 16 × 66.4 µs = 3,928 µs. Just moving the bytes at full speed, with nothing else going on. |
| Best contest kernels | about 6.4 ms | Stock ÷ 2.61, the top score on the whole-layer board when it closed. That score averages six test cases, so treat this as a round figure. |
| Stock SGLang | about 16.6 ms | 46 × 266.9 µs + 16 × 270.4 µs = 16,604 µs, from stock's measured time per layer. More than four times the minimum. |
One user, eight tokens checked per step, GPU work only, as timed by the contest's kernel judge. Talking between GPUs and the rest of the server come on top.
The 12.7 ms between the first and last rows is time the GPU spends doing something other than fetching at full speed: starting and stopping over a thousand small programs per step, workers waiting for each other, and unpacking weights slowly. That gap is what everyone is fighting over.
The only three ways to go faster
Every trick on the plan pages is one of these three wearing a technical name.
| Name on the plan pages | Plain meaning | Which way |
|---|---|---|
| 12-bit codec, 6-bit scales | Pack numbers tighter without changing them. | 1 |
| Fewer launches, fusing, epilogue | Do several small jobs inside one program. | 2 |
| CUDA graph | Record the sequence of programs once and replay it. | 2 |
| PDL chain | Let the next program begin before the last has finished. | 2 |
| Stream-K, split-K, balanced split | Give all 132 workers equal shares. | 3 |
| TMA ring, bytes in flight | Order weights ahead so the road is never empty. | 3 |
Sort these yourself
Click the kind each trick belongs to. Two of them are traps: they would be fast, and the rules forbid them.
Nine to sort.
Which part is worth speeding up?
A step is made of parts: attention, experts, projections, and so on. Making one part faster only helps in proportion to how much of the total it was. This rule is called Amdahl's law, and it decides the order of work on the plan.
new time = untouched share + (sped-up share ÷ its speedup)
Worked example: stock's expert code
On stock, the experts take 141.9 µs of a 266.9 µs layer at one user. That is 53%. The best public expert kernel is 2.4× faster.
- Untouched share: 100% − 53% = 47%.
- Sped-up share: 53% ÷ 2.4 = 22%.
- New time: 47% + 22% = 69% of the old time.
- Overall speedup: 100 ÷ 69 = 1.45×. Not 2.4×.
Three lessons fall out.
- A huge speedup of a small part is worth little.
- There is a ceiling. Even an infinite speedup of a part worth 53% gives at most 1 ÷ 0.47 = 2.1×.
- Once the big part is fast, the shares change, so you measure again before choosing the next target. In one team's already-optimized layer (32 users, one token each) the 154 µs split into experts 84, the four fixed matrices 51.5, attention 9.8 and glue 8.8. Experts were still the largest part, at 55%.
The rules, and how the score is kept
Lossless: repack freely, never round
"Without changing a single answer" has a precise meaning. A change is lossless if every number comes out exactly as it was. A zip file is lossless. A blurry photo is not.
| Allowed | Not allowed |
|---|---|
| Storing weights in a tighter layout that unpacks to the same bits. | Rounding weights to fewer bits. |
| Reordering, merging or overlapping the programs. | Skipping work that "probably" does not matter, such as running 7 of a token's 8 experts. |
| Dealing the same arithmetic out differently across the workers. | Swapping in a rougher, faster formula. |
One subtlety. Adding the same numbers in a different order can change the last digit. So the serving round does not demand identical output. It asks that a changed server agree with stock about as well as stock agrees with itself from run to run. Page 4 explains.
Two rounds, two scoreboards
| Kernel round (closed) | Serving round (open) | |
|---|---|---|
| What is timed | One layer's programs on one GPU, alone. | The whole server on eight GPUs, answering a set of conversations. |
| Score | How many times faster than SGLang's own kernel. | How much more the server puts out at the speed stock gives each user, with 1, 4, 16 and 32 users at once. |
| Limits | Results must pass a check against a reference. | Lossless. The wait before a reply starts may be at most twice stock's. At most 2 GB more memory per GPU. A change of at most 400 KB that SGLang's maintainers would accept. |
Both rounds combine their test cases with a geometric mean, or geomean: multiply the ratios, then take the root. With four ratios of 2.0, 1.5, 1.2 and 1.1, the product is 3.96 and its fourth root is 1.41. A plain average would say 1.45. A geomean is harder on one weak case: 2, 2, 2 and 1 give only 1.68. Page 6 covers the serving score in full.
How to read the plan pages
- µs is a microsecond, a millionth of a second. A layer takes a few hundred. ms is a thousand of those.
- 2.4× means 2.4 times faster than stock SGLang on the same job.
- Floor is the time physics allows: bytes divided by memory speed. No code can beat it without moving fewer bytes.
- b1, b16, b32 are batch sizes: 1, 16 or 32 users at once. d and v mean decode (one token each) and verify (eight tokens each, the check in draft and check). s and f mean a sliding layer, which looks back at the last 2,048 tokens only, or a full layer, which looks back at everything. So "v32s" is verify, 32 users, sliding layer.
- T or m is the number of tokens handled at once.
- CTA is a block: the piece of a kernel that runs on one of the 132 workers.
- Measured means someone timed it on the real GPU. Hypothesis means nobody has yet. EV is expected value: how promising an idea looks before it is tried.
Label decoder
Try it yourself
Pencil and paper
Work each one out before opening the answer. Everything you need is on this page.
1. The pooler table has 4,096 rows and 8,192 columns of 16-bit numbers. How big is it, and what is its floor at 4.3 MB per µs?
4,096 × 8,192 × 2 bytes = 67.1 MB. 67.1 ÷ 4.3 = 15.6 µs.
2. Pack the pooler into 12 bits per number. What are the new size and floor, and what does that save over a 62-layer step?
67.1 × 12 ÷ 16 = 50.3 MB, and 50.3 ÷ 4.3 = 11.7 µs. That saves 3.9 µs per layer, or 62 × 3.9 = 242 µs per step: about a quarter of a millisecond from one table.
3. With one user and 8 tokens per step, stock takes 266.9 µs for a sliding layer and 270.4 µs for a full one. There are 46 sliding and 16 full layers. How long is a step?
46 × 266.9 = 12,277 µs and 16 × 270.4 = 4,326 µs. Together 16,604 µs, which is 16.6 ms.
4. The best kernels take about 6.4 ms per step. If 5 tokens are kept per step, how many tokens per second is that, and how long does a 220-token reply take?
5 tokens ÷ 6.4 ms = 781 tokens per second. The reply needs 44 steps: 44 × 6.4 = 282 ms.
5. In an optimized layer of 154 µs, the experts take 84 µs and attention 9.8 µs. (a) The experts become 1.5× faster. (b) Instead, attention becomes infinitely fast. What is the overall gain in each case?
(a) 70 µs untouched plus 84 ÷ 1.5 = 56 µs makes 126 µs. 154 ÷ 126 = 1.22×. (b) 154 − 9.8 = 144.2 µs. 154 ÷ 144.2 = 1.07×. A modest gain on the big part beats a miracle on the small one.
6. The SGLang team counted about 1,450 kernel launches in one stock step (on a newer GPU). Suppose each wastes 2 µs starting and stopping. How much is that per step, and how does it compare with the 3.9 ms floor?
1,450 × 2 µs = 2.9 ms. That is about three quarters of the floor, spent on no work at all. The 2 µs is a round number for the exercise, but it shows why the second way to go faster matters.
7. A server scores 2.0, 1.5, 1.2 and 1.1 at the four loads. What is its geomean? Then: is it better to lift the 1.1 to 1.32 or the 2.0 to 2.4?
2.0 × 1.5 × 1.2 × 1.1 = 3.96, and the fourth root is 1.41. Both lifts multiply one ratio by 1.2, so both raise the geomean by the same amount, to 1.48. The weak load is usually the easier one to lift.
8. Decode these labels and say how many tokens each handles per step: d16f, v1s, v4s.
d16f: decode, 16 users, full layer, 16 tokens. v1s: verify, 1 user, sliding layer, 8 tokens. v4s: verify, 4 users, sliding layer, 32 tokens.
Feel the warehouse on your own computer
Five minutes on any computer with Python and NumPy. The script adds up a small array many times, then a big array a few times, and reports how many gigabytes per second each went through.
import numpy as np, time
def gb_per_second(n_bytes, repeats):
a = np.ones(n_bytes // 8) # each number takes 8 bytes
a.sum() # warm-up
start = time.perf_counter()
for _ in range(repeats):
a.sum() # read every number once
seconds = time.perf_counter() - start
return n_bytes * repeats / seconds / 1e9
small = gb_per_second(256 * 1024, 20000) # 256 KB: stays in the CPU's cache
big = gb_per_second(1024**3, 5) # 1 GB: hauled from main memory every time
print(f"small array: {small:.1f} GB/s big array: {big:.1f} GB/s")
- What to expect. The arithmetic per byte is identical, yet the small array goes through a few times faster. It fits in the processor's small, fast memory (its cache). The big one has to come down the road from main memory on every pass. On the machine used to test this page the result was about 32 GB/s against 12 GB/s. Yours will differ; the gap is the lesson.
- Compare. An H200's road carries about 4,300 GB/s. It is a very wide road, and the model is still too big for it.
- Go further. Change
256 * 1024to larger sizes and find where the speed drops. That size is roughly your processor's cache.
Optional: feel the launch cost on a GPU
If you have an NVIDIA GPU and PyTorch, this launches 1,000 tiny kernels. Adding eight numbers takes no time at all, so nearly everything it measures is overhead: Python, PyTorch and the launch itself.
import torch, time
x = torch.ones(8, device="cuda")
torch.cuda.synchronize()
start = time.perf_counter()
for _ in range(1000):
x = x + 1 # one tiny kernel each time
torch.cuda.synchronize()
print((time.perf_counter() - start) * 1000, "µs per tiny kernel")
Multiply your result by the 1,450 launches of a stock step to see why servers record and replay their launches. This script was not run for this page, as no GPU was to hand.
Check yourself
Eight questions. Each one gives its reason as soon as you answer. For much more, the practice page has 142 questions, 10 hands-on labs, number drills that never run out and four timed exams.
Five common misreadings
| Tempting thought | What is actually true |
|---|---|
| A chip with faster arithmetic would fix this. | At small batches the limit is the road, not the cooks. |
| A kernel that is 2.4× faster makes the server 2.4× faster. | Only its share of the time shrinks, and a server has costs no kernel touches. |
| Draft and check trades accuracy for speed. | The output is what the big model would have written alone. |
| Serving more users at once is free speed. | The machine puts out more, but each user's reply slows. A step for 32 users takes about three times as long as a step for one. |
| Lossless means the output never differs by a single bit. | The weights must unpack to the same bits. Sums added in a new order may differ in the last digit, and the serving round allows for that. |
- At one user, a step's time is set by fetching weights, not by arithmetic.
- Floor = bytes ÷ road speed. For a step it is about 3.9 ms; stock takes 16.6 ms.
- There are three ways to close that gap: fewer bytes, less waste around the work, every part busy.
- Work on the biggest part first, and measure again after each win.
- Whatever you do, the answers must stay the same.
Study path
The times are rough. Move on from a page when you can do what its last column says.
| Page | What it adds | Time | You are ready to move on when you can |
|---|---|---|---|
| 1 · Start here | The whole picture and the vocabulary. | 30 min | name the three ways to go faster and work out a floor from a size in bytes. |
| 2 · Where a GPU's time goes | Memory as a distant warehouse, the 132 workers, the life of one kernel. With a fetch-versus-arithmetic calculator. | 20 min | explain why a 15 µs kernel loses a large share of its time at its edges. |
| 3 · The model, one layer at a time | Attention and the notes it keeps, the 256 experts, eight GPUs sharing the work, draft and check. | 25 min | list the parts of a layer in order and say which costs most. |
| 4 · Number formats and "lossless" | 16-bit, 8-bit and 4-bit numbers, how 4-bit weights work, how to shrink data without changing it. | 20 min | say why packing 16 bits into 12 is lossless and rounding is not. |
| 5 · The kernel tricks | Each technique named on the plan pages, as problem, idea and catch. | 30 min | match every name in the table above to its trick and its catch. |
| 6 · Serving and the score | Why batching trades speed for volume, what the score measures, and an explorer for it. | 25 min | explain why time that does not grow with users counts for more. |
Test yourself as you go on the practice page: take its quiz for a page once you have read that page, and sit the final exam at the end.
After the primer, read the overview, then the decode-layer page, where the byte table from this page appears in full, then the expert page and the serving page.
Glossary
- Token
- The unit a model reads and writes; about three quarters of a word.
- Weights
- The model's learned numbers. They never change while serving.
- Activations
- The numbers being computed for the current tokens. They change every step.
- Layer
- One of the model's 62 stages. Each looks back at earlier tokens, then updates the current token's numbers.
- Step
- One round of draft and check: the draft guesses 8 tokens and the big model checks them in one pass through all 62 layers.
- GPU, H200
- A chip that does huge amounts of simple arithmetic at once. The H200 is the NVIDIA model this contest runs on.
- SGLang, stock
- The open-source server that runs the model. Stock is SGLang unmodified, the baseline for every comparison.
- Kernel
- A small program that runs on the GPU and does one job, such as multiplying a matrix.
- Launch
- Starting a kernel. Each launch costs a few microseconds before any work happens.
- SM
- Streaming multiprocessor: one of the 132 worker units inside an H200.
- CTA, block
- The piece of a kernel that runs on one SM.
- Warp
- A team of 32 threads on one SM that always act in lockstep.
- Tensor core, wgmma
- The special circuit inside each SM that multiplies small matrices very fast, and the instruction that drives it on an H200.
- TMA
- The H200's built-in machinery for ordering blocks of memory ahead of time.
- HBM
- The GPU's main memory: 141 GB, where the model's weights live.
- Bandwidth
- How fast bytes can leave main memory: about 4.3 trillion bytes a second in practice, which is 4.3 MB per microsecond.
- Floor
- The shortest time physics allows for a job: its bytes divided by the bandwidth.
- Roofline
- A quick speed limit for a job: the larger of the time to fetch its bytes and the time to do its arithmetic.
- L2 cache, shared memory, registers
- Three smaller, faster stores closer to the workers. See page 2.
- GEMV, GEMM
- Matrix times one vector, and matrix times many. Most of the work is one of these.
- Attention
- The part of a layer that looks back at earlier tokens.
- KV cache
- Notes (a key and a value) saved for every earlier token so they are not recomputed.
- Sliding layer, full layer
- 46 of the layers look back only at the last 2,048 tokens. The other 16 look back at the whole conversation.
- GQA
- Grouped-query attention: several questions share one pile of notes, so one read can serve them all.
- MoE, expert, router
- Mixture of experts: 256 small networks per layer, of which a router picks 8 for each token.
- Marlin, W4A16
- Marlin is the expert kernel stock SGLang uses on the H200. W4A16 means 4-bit weights with 16-bit activations.
- Prefill, decode
- Reading the incoming message, and writing the reply.
- Speculative decoding, verify
- A small draft model guesses several tokens and the big model checks them in one pass. The check is called verify.
- DFlash
- The draft model scheme used with this model.
- TP8, EP8
- Two ways the eight GPUs split the model: each holds a slice of every matrix, and each holds 32 of the 256 experts.
- BF16, FP8, NVFP4
- Number formats using 16, 8 and about 4.5 bits per number.
- Lossless
- A change that leaves every result exactly as it was.
- Amdahl's law
- Speeding up one part helps only in proportion to that part's share of the total.
- Batch, b1, b16, b32
- How many users are served in the same step: 1, 16 or 32.
- Judge
- The organizers' program that times an entry and checks its results.
- TTFT
- Time to first token: how long a user waits before the reply starts.
- Per-user speed, throughput
- Tokens per second one user sees, and tokens per second the whole machine produces. The contest calls the first one interactivity.
- Geomean
- An average for ratios: multiply them and take the root. One weak ratio drags it down.
- CUDA graph, PDL
- Two ways to cut launch overhead: replay a recorded sequence of kernels, and let the next kernel start before the last has finished.
- Fusing, epilogue
- Doing small follow-up jobs at the tail of a bigger kernel. The tail is the epilogue.
- Megakernel
- One kernel that never exits and handles a whole layer.
- Stream-K, split-K
- Ways to give every worker an equal share of a multiplication.
- RMSNorm, RoPE
- Two small steps inside a layer: rescaling a token's numbers to a standard size, and marking each token's position. Safe to skip at first.
Longer explainers on this site
- GPU warps and warp shuffle, interactive
- Fused kernels on tensor cores
- Inside the transformer decoder
- Mixture of experts: the field guide
- How LLM quantization works, and its chapter on what a GPU does with 4-bit weights
Numbers on this page come from the plan pages: the per-step and per-layer times from the serving page, the floors and matrix shapes from the decode-layer page, and the model facts, rules and dates from the overview. The launch count is from the SGLang team's megakernel post. Next: where a GPU's time goes.