Serving harness, run budget and pre-submit checklist

About seven scored runs remain, so every decision has to be made on a local copy of the serving judge. This page is that copy, the questions still open about the real one, and the gates to run before spending an official run. Added Nov 14.

How to read this page. The official judge is hidden. The rules here come from the official summary (llms.txt); the workload shape is inferred from the kernel problems and from participants' notes. Where the real judge could differ, the item is listed under open questions. Resolve those from RULES.md and the serving API before trusting a local number. Nothing on this page has been run.

Open questions about the real judge

QuestionWhy it mattersWorking assumption until answered
How is "throughput at stock's per-user speed" computed?At a fixed number of users a faster system has both higher speed and higher throughput. Holding speed at stock's value means moving to more users, which at 32 means going past the ladder.Interpolate your own curve across loads; run 48 users locally to bracket the top load. Under this reading the low-load ratios are strongly amplified (sketch); the first scored run shows whether it holds.
Is per-user speed decode-only, or does it include TTFT?Decides whether faster prefill is rewarded or only gated.Decode-only, with TTFT as a separate 2× gate.
Are users closed-loop (next turn sent when the last one ends)? Is there think time?Sets how much prefill and decode overlap at 16 and 32 users.Closed-loop, no think time.
How are exactly 220 output tokens enforced?Stop handling must match, or throughput differs.A fixed maximum with end-of-sequence ignored.
Which launch flags are fixed?Speculative settings, parallel layout, cache type, request caps and prefill chunking are the cheap levers.Read the flags file before planning any flag work.
Does the board keep the best scored run or the latest?Changes how boldly to use the last runs.The latest. Never submit a scored run you expect to be worse.
What does a check run report?If it returns a per-load ratio, TTFT and agreement, three checks a day can map all four loads in two days.Spend the first check on a no-op or flags-only patch to find out.
Does "lossless" permit the kernel-track numeric allowances?FP16 partial sums, approximate exp2 and FP8-stored P decide the attention design.No.
Is GPU code compiled at start-up from source strings accepted, and is compile time counted?Kernel ports depend on it, and long compiles could hit build limits.Accepted, since no binaries are shipped. Compile once at start-up and keep it short.
How is build memory measured?Peak during load, or resident afterwards? In-place re-encoding has a temporary peak.Peak. Convert layer by layer and free as you go.
What does the agreement check compare?Token match, log-probabilities or task outcome; greedy or sampled.Token-level agreement against stock's own run-to-run band.

Local rig

Workload replica

PropertyReplica settingBasis
ConversationsMulti-turn agentic coding: a long first turn, then short turns that append to a growing history.Official: multi-turn conversations replayed at several loads. Notes: agentic coding.
First turnAbout 16K tokens, prefilled in chunks.Inferred from the prefill problem (16,384-token first-turn chunks).
Later turnsAbout 750 new tokens on a history that grows toward 80K.Inferred from the prefill problem (753 new tokens on an 80K-token conversation).
OutputExactly 220 tokens per request.Notes.
Users1, 4, 16, 32, plus off-ladder 8, 24 and 48.The official ladder. The extras catch tuning to the ladder and bracket the top load for the score proxy.
ContentReal coding-agent transcripts: code, diffs, tool results, file dumps. At least two disjoint trace sets.Speculative-decoding acceptance depends on content, and top entries are rerun on held-out data.

What to record per run

SGLang exposes Prometheus metrics for most of these (TTFT histogram, generation throughput, cache hit rate, running requests). Check the names in the v0.5.23 tree.

Score proxy

A local stand-in for the official score under the working assumptions above. It reads your throughput off your own speed-versus-throughput curve at stock's per-user speed for each load.

import math
LOADS = [1, 4, 16, 32]

def interp_throughput(curve, speed):
    """curve: [(per_user_speed, tokens_per_s_per_gpu)] measured at several loads.
    Log-linear interpolation of throughput at a given per-user speed; None if not bracketed."""
    pts = sorted(curve)                                   # ascending speed = descending load
    for (s0, t0), (s1, t1) in zip(pts, pts[1:]):
        if s0 <= speed <= s1 and s1 > s0:
            w = (math.log(speed) - math.log(s0)) / (math.log(s1) - math.log(s0))
            return math.exp(math.log(t0) + w * (math.log(t1) - math.log(t0)))
    return None

def proxy_score(stock, patched_curve, ttft_stock, ttft_patched):
    """stock: {load: (per_user_speed, throughput)}.
    patched_curve: points at LOADS plus the off-ladder loads (include one above 32)."""
    ratios = {}
    for c in LOADS:
        speed, tput = stock[c]
        if ttft_patched[c] > 2 * ttft_stock[c]:
            ratios[c] = None                              # over the TTFT gate: treat as a failed load
            continue
        t = interp_throughput(patched_curve, speed)
        ratios[c] = t / tput if t else None               # None: run a higher load to bracket it
    ok = [r for r in ratios.values() if r]
    score = math.exp(sum(map(math.log, ok)) / len(ok)) if len(ok) == len(LOADS) else None
    return score, ratios

Report the four ratios, not only the geomean: the lowest ratio is the next thing to work on.

A/B protocol and noise

  1. Write the predicted number and a kill threshold before each run.
  2. Same node, image, flags and traces. Alternate arms with a server restart between: stock, patched, stock, patched.
  3. Discard warm-up. Use the same seeds and the same conversation order in both arms.
  4. At least three repeats per arm for a keep-or-drop decision. The noise band is the spread of stock against stock. Kernel-round judge noise was about 1.5–2% run to run, and serving has more moving parts.
  5. One change per patch. Re-profile after each kept change, because the next bottleneck moves.
  6. Promote only if no load regresses, the off-ladder loads included.

Profiling a stock step

BucketWhat to look forLeads to
Layer kernels by familyExperts, dense projections, attention (sliding and full), norms and glue, per GPU rank.The Tier 2 order on the serving plan.
CommunicationTime and count of collectives per layer; which rank the step waits on.Whether an existing all-reduce fusion is worth switching on.
Draft modelShare of the step; launch count.Draft fusion, only above about a tenth of the step.
Host and gapsTime between graph replays; Python on the step path; sampling and verification on the host.Tier 1 host work.
Prefill shareFraction of GPU time spent on prefill at 16 and 32 users; how long decode stalls when a turn arrives.Interleave policy and chunk size.

Compare the layer-kernel total with the judge-derived estimate on the serving page: about 16.6, 35.2 and 51.1 ms per stock verify step at 1, 16 and 32 users. A large gap in either direction is a finding.

Agreement gate

Pre-submit checklist

okc serve init                      # re-run: the base moved to v0.5.23 (297e54fd) on Nov 11
okc serve diff                      # writes serving/patch.diff
wc -c serving/patch.diff            # keep under about 390,000 bytes
awk '/^\+/ && length($0) > 401' serving/patch.diff | head              # added lines over 400 characters: expect nothing
grep -E '^(rename |Binary files|GIT binary patch)' serving/patch.diff  # renames or binaries: expect nothing
grep '^+++ ' serving/patch.diff | grep -v ' b/python/sglang/'          # outside python/sglang (JSON configs excepted): expect nothing
grep '^+++ ' serving/patch.diff | grep 'srt/model_loader/'             # loader files: expect nothing
nvidia-smi --query-gpu=index,memory.used --format=csv                 # server ready, no traffic: stock vs patched
okc serve check serving/patch.diff  # one of three checks per day; 16 users unless another load is chosen
GatePass condition
AgreementInside stock's band, greedy and sampled, on two trace sets.
TTFTAt or below about 1.6× stock at every load. The limit is 2×.
Build memoryWithin 2 GB per GPU of stock, measured at peak if you can.
Every loadAt or above stock outside noise at 1, 4, 16 and 32, and no regression at 8, 24 or 48.
GeneralityNo branch on the four loads or on trace content. Dispatch by capability and range; anything unsupported falls back to stock's path. Other models and GPUs take the stock path untouched.
StabilityThree cold starts without a hang; a long run without memory growth.
Review shapeA small diff, readable source, tests for new paths, attribution headers on the Apache-2.0 code you build on, no dead variants.

Run-budget ledger

The quota resets at 00:00 UTC, which is 4 pm PT. The track closes at 11:59 pm PT on Nov 19, which is 07:59 UTC on Nov 20. Counting from Nov 14 that is at most 7 scored runs and 21 checks. Plan on 5 and 15: the kernel top 50 go first in the queue, a run can take up to 90 minutes, and it is not stated whether a run queued near the close still counts.

UTC dayWindow in PTChecks (3)Scored run (1)
Nov 14until 4 pm Nov 14One no-op or flags-only check at 16 users, to learn the report format.Hold.
Nov 154 pm Nov 14 → 4 pm Nov 15Tier 0 and 1 at 1 and 32 users.First entry, once all four loads beat stock locally.
Nov 164 pm Nov 15 → 4 pm Nov 16Experts at 16 and 1.Only if better locally at every load.
Nov 174 pm Nov 16 → 4 pm Nov 17The weakest load; 4 users.Same rule.
Nov 184 pm Nov 17 → 4 pm Nov 18Dense projections; the weakest load.Same rule.
Nov 194 pm Nov 18 → 4 pm Nov 19The frozen candidate at two loads.The safe best.
Nov 204 pm → 11:59 pm Nov 19Spare.Spare. Only a clear gain; do not rely on this window.

Running agents on this

Sources: official rules summary (llms.txt: score, loads, limits, CLI); chenwainuo/okc (220 tokens, agreement wording, single-GPU caveat); TASK.md prefill shapes via the prefill page; Quesma top-50 reports (counters, agent process); alygg77 and trxphxle (judge noise). Links on the overview and prior art.