Start here: the challenge in plain words

A primer for readers who have never written GPU code. Six short pages, one idea at a time, with pictures and things to drag. Nothing harder than division. This first page stands on its own: read only this one and you will still have the whole picture.

After this page you can: About 30 minutes with the exercises. You need arithmetic only. One optional experiment uses Python.

On this page The short version Follow one reply Why it is slow Three ways to go faster Which part first Rules and scores Reading the plan pages Try it yourself Check yourself Study path Glossary

The short version

A language model writes one token (roughly a piece of a word) at a time. To produce each token it has to run its whole network, and a network this size does not fit in any fast memory. Its numbers must be hauled in from the GPU's main memory again for every token. Most of the time the expensive calculators inside the GPU sit idle, waiting for numbers to arrive.

The challenge is to make that faster for one model (Meta's Muse Spark Open) on one machine (eight NVIDIA H200 GPUs), without changing a single answer. First by rewriting the small GPU programs that do the work, called kernels. Then by improving the server that feeds them, an open-source project called SGLang.

The castWhat it isNumbers worth remembering
The modelMuse Spark Open, a language model whose learned numbers (its weights) are public.816 billion weights, about 42 billion used for any one token, 62 layers.
The machineOne server with eight H200s. A GPU is a chip built to do huge amounts of simple arithmetic at once.Per GPU: 141 GB of memory, 132 workers, about 4.3 MB delivered per microsecond.
The serverSGLang, the program that takes requests and runs the model. Unmodified SGLang is called stock.Stock is the baseline. Every "2.4×" on this site means 2.4 times faster than stock.
The contestTwo rounds. Kernels: speed up one layer's programs on one GPU. Serving: speed up the whole server on all eight.The kernel round closed on 4 November 2026. The serving round runs from 12 to 19 November.

Follow one reply from start to finish

Conversationso far Prefillread once, take notes One step, repeated until the reply is finished Draft modelguesses 8 tokens Big modelchecks all 8 at once Keep the guesses that match, fix the first miss62 layers on 8 GPUs, every step

Prefill happens once per message. The step repeats dozens of times per reply, so the step is where the time goes.

Three words in the picture need a line each.

Worked example: a 220-token reply

  1. Suppose 5 tokens survive each check. That figure is assumed for illustration; the real one changes with the text.
  2. 220 tokens ÷ 5 per step = 44 steps.
  3. On stock SGLang one step costs about 16.6 ms of GPU work (ms is a thousandth of a second). 44 × 16.6 ms = 730 ms.
  4. 220 tokens in 0.73 seconds is about 300 tokens per second.

Now change the numbers yourself. The three buttons are the three step times this page keeps coming back to.

Jump to
StockYour setting
tokens per second for this one user

This counts the big model's GPU work only, for one user. A real server adds the draft model, the talk between the eight GPUs and its own bookkeeping, so real speeds are lower. The 220-token length is the one participants report for the serving round.

Why it is slow

Picture a kitchen with 132 very fast cooks. The ingredients are in a warehouse down the road, and there is one road. However fast the cooks are, dinner cannot be ready before the trucks arrive.

The cooks are the GPU's 132 workers. The ingredients are the weights. The warehouse is the GPU's main memory, the only place big enough to hold them. The road is memory bandwidth: how many bytes can leave main memory each second. On an H200 the road carries about 4.3 MB every microsecond (µs, a millionth of a second).

shortest possible time = bytes to fetch ÷ road speed

That one division is the most used sum on this site. Its answer is called the floor. No code can beat the floor, however clever, unless it fetches fewer bytes.

Worked example 1: one matrix

Each layer has a table of weights called qkv_gate, with 2,304 rows and 8,192 columns. Each weight is a 16-bit number, which is 2 bytes.

  1. Size: 2,304 × 8,192 × 2 bytes = 37.7 million bytes = 37.7 MB.
  2. Floor: 37.7 MB ÷ 4.3 MB per µs = 8.8 µs.
  3. The arithmetic on those weights, for one token, takes the GPU's calculators about 0.04 µs at their rated speed.

Fetching takes over 200 times longer than calculating. That is what "the calculators sit idle" means, and it is why a faster calculator would not help here.

Worked example 2: one layer, then one step

Here is everything one GPU must fetch to push one token through one layer, with no draft model. Change the road speed, pack the weights tighter or lengthen the conversation, and watch the floor move.

What is fetchedMBµsShare of the layer
One sliding layer One step of 62 layers

Three things to try.

  1. Untick the box. The layer grows by 32.5 MB and 7.6 µs. Over 62 layers that is nearly half a millisecond per step, and ticking it again wins that back without changing any result. This is the first of the three ways to go faster.
  2. Drag the conversation from 2,000 to 200,000 tokens. The step grows by under 0.2 ms. For one user, looking back is cheap. It gets expensive only when many users each bring a long history.
  3. Slide the road speed. Everything scales with it. Bytes and road speed are the only two things in the formula.

Sizes are rows × columns × bytes per number, from the shapes on the decode-layer page; the expert figure is that page's estimate. With the box unticked, the road at its rated 4.8 MB per µs and an 80,000-token conversation, the table lands within a few percent of the organizers' own floor for this case: 40.0 µs for a sliding layer and 2.5 ms for a step.

The whole contest in three numbers

With draft and check, a step carries 8 tokens, not 1. The big tables are fetched once however many tokens there are, but 8 tokens choose more experts between them, so more bytes are needed. The organizers put the floor for that step at 62.3 µs for each of the 46 sliding layers and 66.4 µs for each of the 16 full ones.

Physical minimum3.9 ms Best contest kernels6.4 ms Stock SGLang16.6 ms
WhatTime per stepWhere the number comes from
Physical minimumabout 3.9 ms46 × 62.3 µs + 16 × 66.4 µs = 3,928 µs. Just moving the bytes at full speed, with nothing else going on.
Best contest kernelsabout 6.4 msStock ÷ 2.61, the top score on the whole-layer board when it closed. That score averages six test cases, so treat this as a round figure.
Stock SGLangabout 16.6 ms46 × 266.9 µs + 16 × 270.4 µs = 16,604 µs, from stock's measured time per layer. More than four times the minimum.

One user, eight tokens checked per step, GPU work only, as timed by the contest's kernel judge. Talking between GPUs and the rest of the server come on top.

The 12.7 ms between the first and last rows is time the GPU spends doing something other than fetching at full speed: starting and stopping over a thousand small programs per step, workers waiting for each other, and unpacking weights slowly. That gap is what everyone is fighting over.

The only three ways to go faster

1. Move fewer bytesStore the same numbers in less space, so there is less to haul. Think of vacuum-packing the ingredients: more fit on each truck. The catch: they must come back out exactly the same. Page 4.
2. Waste less time around the workEvery small program has a start-up delay, a wind-down and a gap before the next one. Think of a kitchen that stops dead between courses. Use fewer programs, overlap them, start the next one early. Pages 2 and 5.
3. Keep every part busyA GPU has 132 workers. If work is dealt out unevenly, some finish early and wait for the rest. Think of 100 cooks standing by while 32 plate the last dishes. Deal it evenly. Page 5.

Every trick on the plan pages is one of these three wearing a technical name.

Name on the plan pagesPlain meaningWhich way
12-bit codec, 6-bit scalesPack numbers tighter without changing them.1
Fewer launches, fusing, epilogueDo several small jobs inside one program.2
CUDA graphRecord the sequence of programs once and replay it.2
PDL chainLet the next program begin before the last has finished.2
Stream-K, split-K, balanced splitGive all 132 workers equal shares.3
TMA ring, bytes in flightOrder weights ahead so the road is never empty.3

Sort these yourself

Click the kind each trick belongs to. Two of them are traps: they would be fast, and the rules forbid them.

Nine to sort.

Which part is worth speeding up?

A step is made of parts: attention, experts, projections, and so on. Making one part faster only helps in proportion to how much of the total it was. This rule is called Amdahl's law, and it decides the order of work on the plan.

new time = untouched share + (sped-up share ÷ its speedup)

Worked example: stock's expert code

On stock, the experts take 141.9 µs of a 266.9 µs layer at one user. That is 53%. The best public expert kernel is 2.4× faster.

  1. Untouched share: 100% − 53% = 47%.
  2. Sped-up share: 53% ÷ 2.4 = 22%.
  3. New time: 47% + 22% = 69% of the old time.
  4. Overall speedup: 100 ÷ 69 = 1.45×. Not 2.4×.
Jump to
the part you sped upeverything else
BeforeAfter
1.45×
overall speedup

Three lessons fall out.

The rules, and how the score is kept

Lossless: repack freely, never round

"Without changing a single answer" has a precise meaning. A change is lossless if every number comes out exactly as it was. A zip file is lossless. A blurry photo is not.

AllowedNot allowed
Storing weights in a tighter layout that unpacks to the same bits.Rounding weights to fewer bits.
Reordering, merging or overlapping the programs.Skipping work that "probably" does not matter, such as running 7 of a token's 8 experts.
Dealing the same arithmetic out differently across the workers.Swapping in a rougher, faster formula.

One subtlety. Adding the same numbers in a different order can change the last digit. So the serving round does not demand identical output. It asks that a changed server agree with stock about as well as stock agrees with itself from run to run. Page 4 explains.

Two rounds, two scoreboards

Kernel round (closed)Serving round (open)
What is timedOne layer's programs on one GPU, alone.The whole server on eight GPUs, answering a set of conversations.
ScoreHow many times faster than SGLang's own kernel.How much more the server puts out at the speed stock gives each user, with 1, 4, 16 and 32 users at once.
LimitsResults must pass a check against a reference.Lossless. The wait before a reply starts may be at most twice stock's. At most 2 GB more memory per GPU. A change of at most 400 KB that SGLang's maintainers would accept.

Both rounds combine their test cases with a geometric mean, or geomean: multiply the ratios, then take the root. With four ratios of 2.0, 1.5, 1.2 and 1.1, the product is 3.96 and its fourth root is 1.41. A plain average would say 1.45. A geomean is harder on one weak case: 2, 2, 2 and 1 give only 1.68. Page 6 covers the serving score in full.

How to read the plan pages

Label decoder

Try it yourself

Pencil and paper

Work each one out before opening the answer. Everything you need is on this page.

1. The pooler table has 4,096 rows and 8,192 columns of 16-bit numbers. How big is it, and what is its floor at 4.3 MB per µs?

4,096 × 8,192 × 2 bytes = 67.1 MB. 67.1 ÷ 4.3 = 15.6 µs.

2. Pack the pooler into 12 bits per number. What are the new size and floor, and what does that save over a 62-layer step?

67.1 × 12 ÷ 16 = 50.3 MB, and 50.3 ÷ 4.3 = 11.7 µs. That saves 3.9 µs per layer, or 62 × 3.9 = 242 µs per step: about a quarter of a millisecond from one table.

3. With one user and 8 tokens per step, stock takes 266.9 µs for a sliding layer and 270.4 µs for a full one. There are 46 sliding and 16 full layers. How long is a step?

46 × 266.9 = 12,277 µs and 16 × 270.4 = 4,326 µs. Together 16,604 µs, which is 16.6 ms.

4. The best kernels take about 6.4 ms per step. If 5 tokens are kept per step, how many tokens per second is that, and how long does a 220-token reply take?

5 tokens ÷ 6.4 ms = 781 tokens per second. The reply needs 44 steps: 44 × 6.4 = 282 ms.

5. In an optimized layer of 154 µs, the experts take 84 µs and attention 9.8 µs. (a) The experts become 1.5× faster. (b) Instead, attention becomes infinitely fast. What is the overall gain in each case?

(a) 70 µs untouched plus 84 ÷ 1.5 = 56 µs makes 126 µs. 154 ÷ 126 = 1.22×. (b) 154 − 9.8 = 144.2 µs. 154 ÷ 144.2 = 1.07×. A modest gain on the big part beats a miracle on the small one.

6. The SGLang team counted about 1,450 kernel launches in one stock step (on a newer GPU). Suppose each wastes 2 µs starting and stopping. How much is that per step, and how does it compare with the 3.9 ms floor?

1,450 × 2 µs = 2.9 ms. That is about three quarters of the floor, spent on no work at all. The 2 µs is a round number for the exercise, but it shows why the second way to go faster matters.

7. A server scores 2.0, 1.5, 1.2 and 1.1 at the four loads. What is its geomean? Then: is it better to lift the 1.1 to 1.32 or the 2.0 to 2.4?

2.0 × 1.5 × 1.2 × 1.1 = 3.96, and the fourth root is 1.41. Both lifts multiply one ratio by 1.2, so both raise the geomean by the same amount, to 1.48. The weak load is usually the easier one to lift.

8. Decode these labels and say how many tokens each handles per step: d16f, v1s, v4s.

d16f: decode, 16 users, full layer, 16 tokens. v1s: verify, 1 user, sliding layer, 8 tokens. v4s: verify, 4 users, sliding layer, 32 tokens.

Feel the warehouse on your own computer

Five minutes on any computer with Python and NumPy. The script adds up a small array many times, then a big array a few times, and reports how many gigabytes per second each went through.

import numpy as np, time

def gb_per_second(n_bytes, repeats):
    a = np.ones(n_bytes // 8)            # each number takes 8 bytes
    a.sum()                              # warm-up
    start = time.perf_counter()
    for _ in range(repeats):
        a.sum()                          # read every number once
    seconds = time.perf_counter() - start
    return n_bytes * repeats / seconds / 1e9

small = gb_per_second(256 * 1024, 20000)   # 256 KB: stays in the CPU's cache
big = gb_per_second(1024**3, 5)             # 1 GB: hauled from main memory every time
print(f"small array: {small:.1f} GB/s    big array: {big:.1f} GB/s")

Optional: feel the launch cost on a GPU

If you have an NVIDIA GPU and PyTorch, this launches 1,000 tiny kernels. Adding eight numbers takes no time at all, so nearly everything it measures is overhead: Python, PyTorch and the launch itself.

import torch, time

x = torch.ones(8, device="cuda")
torch.cuda.synchronize()
start = time.perf_counter()
for _ in range(1000):
    x = x + 1                            # one tiny kernel each time
torch.cuda.synchronize()
print((time.perf_counter() - start) * 1000, "µs per tiny kernel")

Multiply your result by the 1,450 launches of a stock step to see why servers record and replay their launches. This script was not run for this page, as no GPU was to hand.

Check yourself

Eight questions. Each one gives its reason as soon as you answer. For much more, the practice page has 142 questions, 10 hands-on labs, number drills that never run out and four timed exams.

Five common misreadings

Tempting thoughtWhat is actually true
A chip with faster arithmetic would fix this.At small batches the limit is the road, not the cooks.
A kernel that is 2.4× faster makes the server 2.4× faster.Only its share of the time shrinks, and a server has costs no kernel touches.
Draft and check trades accuracy for speed.The output is what the big model would have written alone.
Serving more users at once is free speed.The machine puts out more, but each user's reply slows. A step for 32 users takes about three times as long as a step for one.
Lossless means the output never differs by a single bit.The weights must unpack to the same bits. Sums added in a new order may differ in the last digit, and the serving round allows for that.
What to take away.

Study path

The times are rough. Move on from a page when you can do what its last column says.

PageWhat it addsTimeYou are ready to move on when you can
1 · Start hereThe whole picture and the vocabulary.30 minname the three ways to go faster and work out a floor from a size in bytes.
2 · Where a GPU's time goesMemory as a distant warehouse, the 132 workers, the life of one kernel. With a fetch-versus-arithmetic calculator.20 minexplain why a 15 µs kernel loses a large share of its time at its edges.
3 · The model, one layer at a timeAttention and the notes it keeps, the 256 experts, eight GPUs sharing the work, draft and check.25 minlist the parts of a layer in order and say which costs most.
4 · Number formats and "lossless"16-bit, 8-bit and 4-bit numbers, how 4-bit weights work, how to shrink data without changing it.20 minsay why packing 16 bits into 12 is lossless and rounding is not.
5 · The kernel tricksEach technique named on the plan pages, as problem, idea and catch.30 minmatch every name in the table above to its trick and its catch.
6 · Serving and the scoreWhy batching trades speed for volume, what the score measures, and an explorer for it.25 minexplain why time that does not grow with users counts for more.

Test yourself as you go on the practice page: take its quiz for a page once you have read that page, and sit the final exam at the end.

After the primer, read the overview, then the decode-layer page, where the byte table from this page appears in full, then the expert page and the serving page.

Glossary

Token
The unit a model reads and writes; about three quarters of a word.
Weights
The model's learned numbers. They never change while serving.
Activations
The numbers being computed for the current tokens. They change every step.
Layer
One of the model's 62 stages. Each looks back at earlier tokens, then updates the current token's numbers.
Step
One round of draft and check: the draft guesses 8 tokens and the big model checks them in one pass through all 62 layers.
GPU, H200
A chip that does huge amounts of simple arithmetic at once. The H200 is the NVIDIA model this contest runs on.
SGLang, stock
The open-source server that runs the model. Stock is SGLang unmodified, the baseline for every comparison.
Kernel
A small program that runs on the GPU and does one job, such as multiplying a matrix.
Launch
Starting a kernel. Each launch costs a few microseconds before any work happens.
SM
Streaming multiprocessor: one of the 132 worker units inside an H200.
CTA, block
The piece of a kernel that runs on one SM.
Warp
A team of 32 threads on one SM that always act in lockstep.
Tensor core, wgmma
The special circuit inside each SM that multiplies small matrices very fast, and the instruction that drives it on an H200.
TMA
The H200's built-in machinery for ordering blocks of memory ahead of time.
HBM
The GPU's main memory: 141 GB, where the model's weights live.
Bandwidth
How fast bytes can leave main memory: about 4.3 trillion bytes a second in practice, which is 4.3 MB per microsecond.
Floor
The shortest time physics allows for a job: its bytes divided by the bandwidth.
Roofline
A quick speed limit for a job: the larger of the time to fetch its bytes and the time to do its arithmetic.
L2 cache, shared memory, registers
Three smaller, faster stores closer to the workers. See page 2.
GEMV, GEMM
Matrix times one vector, and matrix times many. Most of the work is one of these.
Attention
The part of a layer that looks back at earlier tokens.
KV cache
Notes (a key and a value) saved for every earlier token so they are not recomputed.
Sliding layer, full layer
46 of the layers look back only at the last 2,048 tokens. The other 16 look back at the whole conversation.
GQA
Grouped-query attention: several questions share one pile of notes, so one read can serve them all.
MoE, expert, router
Mixture of experts: 256 small networks per layer, of which a router picks 8 for each token.
Marlin, W4A16
Marlin is the expert kernel stock SGLang uses on the H200. W4A16 means 4-bit weights with 16-bit activations.
Prefill, decode
Reading the incoming message, and writing the reply.
Speculative decoding, verify
A small draft model guesses several tokens and the big model checks them in one pass. The check is called verify.
DFlash
The draft model scheme used with this model.
TP8, EP8
Two ways the eight GPUs split the model: each holds a slice of every matrix, and each holds 32 of the 256 experts.
BF16, FP8, NVFP4
Number formats using 16, 8 and about 4.5 bits per number.
Lossless
A change that leaves every result exactly as it was.
Amdahl's law
Speeding up one part helps only in proportion to that part's share of the total.
Batch, b1, b16, b32
How many users are served in the same step: 1, 16 or 32.
Judge
The organizers' program that times an entry and checks its results.
TTFT
Time to first token: how long a user waits before the reply starts.
Per-user speed, throughput
Tokens per second one user sees, and tokens per second the whole machine produces. The contest calls the first one interactivity.
Geomean
An average for ratios: multiply them and take the root. One weak ratio drags it down.
CUDA graph, PDL
Two ways to cut launch overhead: replay a recorded sequence of kernels, and let the next kernel start before the last has finished.
Fusing, epilogue
Doing small follow-up jobs at the tail of a bigger kernel. The tail is the epilogue.
Megakernel
One kernel that never exits and handles a whole layer.
Stream-K, split-K
Ways to give every worker an equal share of a multiplication.
RMSNorm, RoPE
Two small steps inside a layer: rescaling a token's numbers to a standard size, and marking each token's position. Safe to skip at first.

Longer explainers on this site

Numbers on this page come from the plan pages: the per-step and per-layer times from the serving page, the floors and matrix shapes from the decode-layer page, and the model facts, rules and dates from the overview. The launch count is from the SGLang team's megakernel post. Next: where a GPU's time goes.