weili-0234/oprefill-hicache-2p1d-glm52-edgebench-ic1

Optimistic Prefill × HiCache 2P1D Ablation — full experiment artifacts

This dataset contains the complete artifacts (client traces, router receipts, engine telemetry/logs, per-request exports, analysis scripts, and results) of a matched-pair ablation of SGLang optimistic prefill in a PD-disaggregated deployment, run on 2026-08-01 on Together's research-b200-ic1 cluster. It is published gated so results can be re-analyzed later without cluster access.

What is NOT here (by confidentiality policy): the private SGLang runtime source bundle (referenced only by commit/SHA pins), and the protected EdgeBench bit-exact corpus (recorded prompt/output token IDs). All lifecycle records use hashed request / trace IDs and contain no prompt or response content.

1. The experiment I ran

Question: does --optimistic-prefill-attempts 1 (prefill starts computing before the decode worker has allocated destination KV; computed KV is withheld until decode signals ready) improve TTFT/E2E for a real agentic workload, at what decode-pressure regimes, and at what cost?

Fixed apparatus (identical in every cell):

• —Model GLM-5.2-FP8 (revision ba978f7d347eaf65d22f1a86833408afdb953541), 2P1D fleet on 3× 8×B200 nodes: P0/P1 = TP8 + PCP8 (--enable-prefill-cp, DSA CP-shared KV), D0 = TP8/DCP1. FP8 E4M3 KV, page 64, mem-fraction-static 0.8 (device pool 949,248 tokens/rank), --hicache-size 100 (≈2.23M host tokens/rank), decode radix cache ON, EAGLE 5-step (top-k 1, 6 draft tokens).
• —Router: sgl-model-gateway compiled from the pinned bundle, retries disabled (one attempt total; 120 s prefill response-header timeout).
• —Private runtime pin: commit 09f74119bda699e96f2e0f16aa975082266eb0b4, bundle SHA-256 a7cd8b08d65bf58a25102c482120fe19c26377ad614a3ea1c9c31c5d38166e76, plus two overlay patches (prefill CP-shared HiCache barrier; decode metrics-reporter fix). Public base image docker.io/weili0234/sglang@sha256:02fc722b....
• —Workload: EdgeBench bit-exact strict replay — 137 recorded Claude-Code agent sessions (17,968 explicit-token requests) replayed with recorded think-time gaps (delay_scale=1), forced recorded output tokens, session cache salt, seed 20260730, corpus cycled to hold the cell's session count constant.

Matrix (counterbalanced A/B, B/A, A/B): 6 cells = {24, 32, 40} sessions × optimistic {0, 1}; each cell = ~6 min tokenization prep + exactly 3600 s admission + ≤900 s drain, with a verified cache-clean transition (flush + per-rank flag check) between cells on the same live fleet — no engine restarts.

How it was executed: a manual controller pod (spec: scripts/manifests/cc1-pod.json) ran each cell via scripts/controller/runcell.sh (per-cell: transition → runtime preflight → metrics sampler (5 s, pod-local spool) → Trie strict-replay client → client/lifecycle/metrics checkers → finalizer). scripts/controller/phase1.sh is the one-time gate (corpus hash verify → baseline preflight → 39-request smoke → router stability). Full flag-level configuration of every cell is inside each cell's cell-config.json and experiment-config receipts in this dataset.

Validity policy (important for interpretation): science gates were enforced unchanged — full 3600+900 s window, pinned config/process-epoch identity, strict-replay exit 0, ≤5% router-confirmed prefill-header timeouts. Observability-side perfection gates (torn multiprocess Prometheus scrapes, metric-fetch CurlExit28 timeouts under load, router health-failure counter ticks, drain-boundary lifecycle correlation) were downgraded from run-invalidating assertions to recorded annotations — see scripts/tolerant-tooling/ for the exact patched checkers and each cell's cc1-annotations.jsonl + telemetry receipts for what fired. All six cells passed the science gates on their first attempt (confirmed timeouts: 1/1/1/1/0/0 out of ~3,500-3,800 admitted requests per cell).

2. The results I got

2.1 Latency (script: scripts/analysis/settlement.py, sign test = on-arm wins

across twelve 300 s admission windows; win_detail.py prints per-window detail)

2.2 Mechanism (scripts: pair_compare.py, kv_breakdown.py, putil2.py,

extract_kv_timeseries.py + plot_kv_composition.py → kv-composition-cell01-cell02.png)

• —Decode KV allocation wait is unchanged by the treatment in every pair and scales brutally with concurrency: 6.6 s (c24) → ~15 s (c32) → ~26 s (c40) mean. The mechanism overlaps prefill compute under this wait; it never shortens it.
• —Prefill retries (sglang:num_prefill_retries_total) = 0 in all six cells — the optimistic yield/requeue cost path never fired.
• —Decode pool composition (see PNG): ~78-85% held by active decode, prealloc-new ≈ 0% (99% of preallocation is radix-prefix references), evictable cache ~6%, free ~15%.
• —Prefill is ~97% cache-served (≈210k prompt TPM/GPU submitted vs ≈6.5k uncached TPM/GPU computed) and prefill GPUs idle at ~20% mean in both arms.

2.3 Throughput (scripts: total_avg.py, tail_throughput.py,

token_throughput.py, windowed_tables.py, cell_durations.py)

• —Admission-window total decode TPS (avg): c24 816/817, c32 863/866, c40 910/898 (off/on) — treatment does not change decode capacity (closed-loop workload).
• —Last-30-min pace: c32 on-arm +9.8% completions/min and +4.6% decode TPS (its faster TTFT compounds); c40 on-arm −2.7% / −3.8% (late-window degradation, cause not yet isolated — leading hypothesis is prefill-side KV residency from early-computed requests pressuring the prefill radix cache over the hour).
• —Per-stream decode TPS p50 ≈ 96-108 tok/s in every cell and both arms (p90/p95 of per-stream TPS are short-output artifacts; trust p25-p70).
• —Every cell: 60.0 min admission exactly, ~20.7 min drain+publish, ~3 min checkers.

2.4 Conclusion

Optimistic prefill is effectively free at this workload (no retries, no decode-side change) and buys meaningful latency in the mid-pressure regime — peak at c32 (−7.5% TTFT p50, −8.4% p90, consistent across windows) — but the benefit drowns at c40 where a ~1 s hidden prefill compute is noise against 26 s allocation waits, and the c24 gain is within single-run noise. p99 movements are inconsistent in sign across pairs (single-run tail noise); replicate a pair if tails matter.

3. File layout

cells/<cell-id>/                      # the six valid cells (1.3-1.5 GB each)
  timed-result.json                   #   Trie client aggregate result
  lifecycle/client.jsonl              #   per-request client events (ADMIT/FIRST_TOKEN/COMPLETE + token counts)
  lifecycle-events.jsonl              #   correlated worker-side events (P_RECEIVE, bootstrap queue, D stages)
  logs/timed-replay.log               #   full Trie client log
  metrics/{p0,p1,d0,router}.openmetrics  # 5 s Prometheus scrapes for the whole cell
  metrics/sampling-control.jsonl      #   scrape-cadence control stream (+errors.jsonl)
  preflight/*.json                    #   transition/runtime/metrics/lifecycle/client gates
  cell-config.json, corpus-receipt.json, transition-receipt.json,
  process-epochs.json, artifact-manifest.json, phase-markers.jsonl
  cc1-annotations.jsonl               #   annotate-only findings for this cell
engine/renamed-workers-r5/            # engine-side artifacts (whole fleet lifetime, incl. prior NR attempts)
  workers/{p0,p1,d0}/requests/        #   per-request engine exports (no bodies)
  workers/{p0,p1,d0}/server.log       #   engine logs
  workers/{p0,p1,d0}/gpu-util.csv     #   1 Hz nvidia-smi utilization
  forward/                            #   forward-pass telemetry shards
  control/, recovery/, post-schedule/ #   controller context + receipts
router/ablation-2p1d-noretry-r1/      # router logs/receipts (confirmed-timeout source)
replay-attempts/                      # archived invalid attempts (NR-era provenance + cc1 receipts)
scripts/                              # controller, analysis, tolerant tooling, pod manifests
kv-composition-cell01-cell02.png      # decode KV pool composition figure (c24 pair)

4. Reproducing the analysis

Every table above regenerates from cells/ alone. Set the cells root, then e.g.:

python scripts/analysis/settlement.py        # latency tables + sign tests
python scripts/analysis/windowed_tables.py   # last-30/60-min throughput tables
python scripts/analysis/token_throughput.py  # TPM/GPU + total TPS + per-stream TPS

(The scripts read RESULT_ROOT from env in their original form — point root at your local cells/ download.)

5. Provenance & caveats

• —Executed by a manual controller after an earlier automated controller spent 25 attempts failing its own observability gates on this matrix; the gate-vs-science separation above is the fix. Cell 01's data was collected 2026-08-01 ~06:00 UTC; cells 02-06 ran 19:45-03:07 UTC (same fleet, zero worker restarts throughout — process-epoch receipts prove identity).
• —Closed-loop strict replay: throughput numbers are pacing, not capacity ceilings.
• —Engine requests/ and server.log span the fleet's full 21 h lifetime (including pre-takeover attempts); filter by the cell time windows in phase-markers.jsonl when correlating.
• —The controller pod's wrapper logs (/runtime/cell0*.log) were lost with the pod; the per-cell Trie logs in cells/*/logs/ carry the same client-side content.