Serving harness, run budget and pre-submit checklist
About seven scored runs remain, so every decision has to be made on a local copy of the serving judge. This page is that copy, the questions still open about the real one, and the gates to run before spending an official run. Added Nov 14.
RULES.md and the serving API before trusting a local number. Nothing on this page has been run.Open questions about the real judge
| Question | Why it matters | Working assumption until answered |
|---|---|---|
| How is "throughput at stock's per-user speed" computed? | At a fixed number of users a faster system has both higher speed and higher throughput. Holding speed at stock's value means moving to more users, which at 32 means going past the ladder. | Interpolate your own curve across loads; run 48 users locally to bracket the top load. Under this reading the low-load ratios are strongly amplified (sketch); the first scored run shows whether it holds. |
| Is per-user speed decode-only, or does it include TTFT? | Decides whether faster prefill is rewarded or only gated. | Decode-only, with TTFT as a separate 2× gate. |
| Are users closed-loop (next turn sent when the last one ends)? Is there think time? | Sets how much prefill and decode overlap at 16 and 32 users. | Closed-loop, no think time. |
| How are exactly 220 output tokens enforced? | Stop handling must match, or throughput differs. | A fixed maximum with end-of-sequence ignored. |
| Which launch flags are fixed? | Speculative settings, parallel layout, cache type, request caps and prefill chunking are the cheap levers. | Read the flags file before planning any flag work. |
| Does the board keep the best scored run or the latest? | Changes how boldly to use the last runs. | The latest. Never submit a scored run you expect to be worse. |
| What does a check run report? | If it returns a per-load ratio, TTFT and agreement, three checks a day can map all four loads in two days. | Spend the first check on a no-op or flags-only patch to find out. |
| Does "lossless" permit the kernel-track numeric allowances? | FP16 partial sums, approximate exp2 and FP8-stored P decide the attention design. | No. |
| Is GPU code compiled at start-up from source strings accepted, and is compile time counted? | Kernel ports depend on it, and long compiles could hit build limits. | Accepted, since no binaries are shipped. Compile once at start-up and keep it short. |
| How is build memory measured? | Peak during load, or resident afterwards? In-place re-encoding has a temporary peak. | Peak. Convert layer by layer and free as you go. |
| What does the agreement check compare? | Token match, log-probabilities or task outcome; greedy or sampled. | Token-level agreement against stock's own run-to-run band. |
Local rig
- Same everything: one 8×H200 SXM node, image
lmsysorg/sglang:v0.5.23, commit297e54fd, the launch flags thatokc serve initwrites, the NVFP4 checkpoint and the DFlash draft. H100 results did not carry over for the kernels. - Two machines if possible: one H200 for kernel-level A/B on a single rank, and the 8-GPU node by the hour for serving runs. A participant's notes make the same split and warn that single-GPU speedups are not serving scores.
- Profiling counters: Nebius, AWS and Google Cloud expose them; RunPod and Vast.ai block them. Without counters, use in-kernel timestamps as on the methodology page.
- Keep the rig's code human-owned, and freeze it once stock against stock repeats within about 2%.
Workload replica
| Property | Replica setting | Basis |
|---|---|---|
| Conversations | Multi-turn agentic coding: a long first turn, then short turns that append to a growing history. | Official: multi-turn conversations replayed at several loads. Notes: agentic coding. |
| First turn | About 16K tokens, prefilled in chunks. | Inferred from the prefill problem (16,384-token first-turn chunks). |
| Later turns | About 750 new tokens on a history that grows toward 80K. | Inferred from the prefill problem (753 new tokens on an 80K-token conversation). |
| Output | Exactly 220 tokens per request. | Notes. |
| Users | 1, 4, 16, 32, plus off-ladder 8, 24 and 48. | The official ladder. The extras catch tuning to the ladder and bracket the top load for the score proxy. |
| Content | Real coding-agent transcripts: code, diffs, tool results, file dumps. At least two disjoint trace sets. | Speculative-decoding acceptance depends on content, and top entries are rerun on held-out data. |
What to record per run
- Per load: TTFT (median and p95), per-user decode speed (tokens after the first ÷ time after the first), and total output tokens/s ÷ 8.
- Per step: step time, accepted tokens per verify step, running batch size.
- Prefix-cache hit rate and queue depth.
- Memory per GPU once the server is ready, before traffic.
- Errors, timeouts and any request that did not return 220 tokens.
SGLang exposes Prometheus metrics for most of these (TTFT histogram, generation throughput, cache hit rate, running requests). Check the names in the v0.5.23 tree.
Score proxy
A local stand-in for the official score under the working assumptions above. It reads your throughput off your own speed-versus-throughput curve at stock's per-user speed for each load.
import math
LOADS = [1, 4, 16, 32]
def interp_throughput(curve, speed):
"""curve: [(per_user_speed, tokens_per_s_per_gpu)] measured at several loads.
Log-linear interpolation of throughput at a given per-user speed; None if not bracketed."""
pts = sorted(curve) # ascending speed = descending load
for (s0, t0), (s1, t1) in zip(pts, pts[1:]):
if s0 <= speed <= s1 and s1 > s0:
w = (math.log(speed) - math.log(s0)) / (math.log(s1) - math.log(s0))
return math.exp(math.log(t0) + w * (math.log(t1) - math.log(t0)))
return None
def proxy_score(stock, patched_curve, ttft_stock, ttft_patched):
"""stock: {load: (per_user_speed, throughput)}.
patched_curve: points at LOADS plus the off-ladder loads (include one above 32)."""
ratios = {}
for c in LOADS:
speed, tput = stock[c]
if ttft_patched[c] > 2 * ttft_stock[c]:
ratios[c] = None # over the TTFT gate: treat as a failed load
continue
t = interp_throughput(patched_curve, speed)
ratios[c] = t / tput if t else None # None: run a higher load to bracket it
ok = [r for r in ratios.values() if r]
score = math.exp(sum(map(math.log, ok)) / len(ok)) if len(ok) == len(LOADS) else None
return score, ratios
Report the four ratios, not only the geomean: the lowest ratio is the next thing to work on.
A/B protocol and noise
- Write the predicted number and a kill threshold before each run.
- Same node, image, flags and traces. Alternate arms with a server restart between: stock, patched, stock, patched.
- Discard warm-up. Use the same seeds and the same conversation order in both arms.
- At least three repeats per arm for a keep-or-drop decision. The noise band is the spread of stock against stock. Kernel-round judge noise was about 1.5–2% run to run, and serving has more moving parts.
- One change per patch. Re-profile after each kept change, because the next bottleneck moves.
- Promote only if no load regresses, the off-ladder loads included.
Profiling a stock step
| Bucket | What to look for | Leads to |
|---|---|---|
| Layer kernels by family | Experts, dense projections, attention (sliding and full), norms and glue, per GPU rank. | The Tier 2 order on the serving plan. |
| Communication | Time and count of collectives per layer; which rank the step waits on. | Whether an existing all-reduce fusion is worth switching on. |
| Draft model | Share of the step; launch count. | Draft fusion, only above about a tenth of the step. |
| Host and gaps | Time between graph replays; Python on the step path; sampling and verification on the host. | Tier 1 host work. |
| Prefill share | Fraction of GPU time spent on prefill at 16 and 32 users; how long decode stalls when a turn arrives. | Interleave policy and chunk size. |
Compare the layer-kernel total with the judge-derived estimate on the serving page: about 16.6, 35.2 and 51.1 ms per stock verify step at 1, 16 and 32 users. A large gap in either direction is a finding.
Agreement gate
- Band: run stock twice on the same traces and measure how often outputs match token for token, greedy and sampled with fixed seeds.
- Test: patched against stock must fall inside that band.
- Module level: for each replaced module, capture real inputs from a stock run and compare outputs on the same inputs. A lossless re-encoding should differ only by summation order; anything larger is a bug or a precision change.
- Router: on captured hidden states, the set of 8 experts per token must be identical to stock's. A different pick is a different model output.
- Structure: all 8 experts computed, the full attention window, DFlash on, 220 tokens out.
Pre-submit checklist
okc serve init # re-run: the base moved to v0.5.23 (297e54fd) on Nov 11
okc serve diff # writes serving/patch.diff
wc -c serving/patch.diff # keep under about 390,000 bytes
awk '/^\+/ && length($0) > 401' serving/patch.diff | head # added lines over 400 characters: expect nothing
grep -E '^(rename |Binary files|GIT binary patch)' serving/patch.diff # renames or binaries: expect nothing
grep '^+++ ' serving/patch.diff | grep -v ' b/python/sglang/' # outside python/sglang (JSON configs excepted): expect nothing
grep '^+++ ' serving/patch.diff | grep 'srt/model_loader/' # loader files: expect nothing
nvidia-smi --query-gpu=index,memory.used --format=csv # server ready, no traffic: stock vs patched
okc serve check serving/patch.diff # one of three checks per day; 16 users unless another load is chosen
| Gate | Pass condition |
|---|---|
| Agreement | Inside stock's band, greedy and sampled, on two trace sets. |
| TTFT | At or below about 1.6× stock at every load. The limit is 2×. |
| Build memory | Within 2 GB per GPU of stock, measured at peak if you can. |
| Every load | At or above stock outside noise at 1, 4, 16 and 32, and no regression at 8, 24 or 48. |
| Generality | No branch on the four loads or on trace content. Dispatch by capability and range; anything unsupported falls back to stock's path. Other models and GPUs take the stock path untouched. |
| Stability | Three cold starts without a hang; a long run without memory growth. |
| Review shape | A small diff, readable source, tests for new paths, attribution headers on the Apache-2.0 code you build on, no dead variants. |
Run-budget ledger
The quota resets at 00:00 UTC, which is 4 pm PT. The track closes at 11:59 pm PT on Nov 19, which is 07:59 UTC on Nov 20. Counting from Nov 14 that is at most 7 scored runs and 21 checks. Plan on 5 and 15: the kernel top 50 go first in the queue, a run can take up to 90 minutes, and it is not stated whether a run queued near the close still counts.
| UTC day | Window in PT | Checks (3) | Scored run (1) |
|---|---|---|---|
| Nov 14 | until 4 pm Nov 14 | One no-op or flags-only check at 16 users, to learn the report format. | Hold. |
| Nov 15 | 4 pm Nov 14 → 4 pm Nov 15 | Tier 0 and 1 at 1 and 32 users. | First entry, once all four loads beat stock locally. |
| Nov 16 | 4 pm Nov 15 → 4 pm Nov 16 | Experts at 16 and 1. | Only if better locally at every load. |
| Nov 17 | 4 pm Nov 16 → 4 pm Nov 17 | The weakest load; 4 users. | Same rule. |
| Nov 18 | 4 pm Nov 17 → 4 pm Nov 18 | Dense projections; the weakest load. | Same rule. |
| Nov 19 | 4 pm Nov 18 → 4 pm Nov 19 | The frozen candidate at two loads. | The safe best. |
| Nov 20 | 4 pm → 11:59 pm Nov 19 | Spare. | Spare. Only a clear gain; do not rely on this window. |
Running agents on this
- A person must accept the rules through
okc login; agents cannot. Give sandboxes a token fromokc api token. - Use a fresh worker per attempt and one integrator that merges measured wins. Long-lived agents stalled in the kernel round.
- Agents reward-hack harnesses. Keep the rig and the gates outside their write access, and have a person press the button on official runs: there is one scored run a day. Runs that gamed the judge can be withdrawn (
okc withdraw). - Every attempt records its prediction, its result at each load and its patch size.
Sources: official rules summary (llms.txt: score, loads, limits, CLI); chenwainuo/okc (220 tokens, agreement wording, single-GPU caveat); TASK.md prefill shapes via the prefill page; Quesma top-50 reports (counters, agent process); alygg77 and trxphxle (judge noise). Links on the overview and prior art.