SoroushVahidi/llm-serving-scheduler-portability

LLM-Serving Scheduler Portability (LSSP) — v1.0.0

LSSP measures how portable comparative LLM-serving scheduler rankings are across workload sources, load regions, evaluation metrics, and SLO definitions: does the best-performing scheduling policy stay best when you change the traffic source or the load level, or do rankings reverse? This dataset is the derived analysis behind that question, paired with a selected-case physical validation (RQ6).

The underlying simulator campaign spans 9,360 unique policy–window–region configurations, each executed in two deterministic verification passes (18,720 total executions); the workload window, not the verification repetition, is the inferential unit.

This is an initial (v1.0.0), intentionally partial release — see "What's in this release" below for exactly what is and is not included, and why. Everything included is either byte-identical to its source (noted per file) or an explicitly-labeled, content-complete-on-the-claims- that-matter reduction.

Relation to the paper

Companion dataset to "How Portable Are LLM-Serving Scheduler Rankings Across Workloads, Operating Regions, and Metrics?" (Soroush Vahidi; GitHub: `SoroushVahidi/llm-serving-scheduler-robustness-benchmark`). This dataset revision corresponds to GitHub release `v1.0.0` (commit 18128a8cf4d449c333c6db4d31788dd5eae180bd).

What's in this release

Not included in this v1.0.0 revision, and why:

• —`table_data/rq3_reversals.json`, `table_data/rq5_temporal_robustness.json` and the *six intermediate canonical `analysis_canonical/.json outputs** (pairwise reversals, ranking correlations, sample complexity, telemetry explanation, temporal robustness, top-k overlap): deferred to a follow-up dataset revision. The manuscript's headline RQ1/RQ2/RQ4/RQ5 numbers are already fully present above; the deferred files back RQ3's pilot-scale result and finer per-metric/per-condition detail beyond the headline tables. Because these six files are absent, paper/scripts/generatephase12tablesfigures.py` (which consumes them as input) cannot currently be run against this dataset download — the released `tabledata/.json` files are that script's output*, already provided directly.
• —`manifests/` (frozen campaign manifests, including the 18,720-execution campaign freeze): not duplicated here — every one of these files is already publicly tracked in the GitHub repository under artifacts/manifests/ in the tagged v1.0.0 release.
• —The full 240 raw per-cell RQ6 execution records and the exploded per-cell raw/enriched shards for RQ1–RQ5 (tens of megabytes, not yet packaged for release): the included reduced/canonical files are sufficient to verify every reported claim; the raw per-cell shards exist only for independent verification of the analysis step itself and are candidates for a follow-up revision.
• —Raw third-party workload traces (BurstGPT, Azure LLM Inference Trace 2024, Bailian/Qwen): not redistributed by this project under any revision; obtain from their original sources subject to their respective terms — see LSSP_THIRD_PARTY_SOURCE_LICENSES.md.

RQ3 status

RQ3 (synthetic-to-real ranking transfer) is an engineering-validation pilot only. It does not support a synthetic-to-real transfer conclusion, positive or negative; the transfer statistic is undefined in most pilot conditions by design. See the manuscript's Results section for detail.

RQ6 status

RQ6 (Slurm job 1222413, execution SHA 703a752762348bd911c9d93f17731fa5244b38f9) is complete: 240/240 cells COMPLETED, independently validated, and analyzed with the frozen robustbench.real_llm.rq6_validation_analysis implementation (2,000-resample window-level paired bootstrap, 95% CI). Result: all three real-system slai_faithful-vs-vllm_faithful ANWG effects are statistically supported and favor vllm_faithful; the simulator-predicted Azure/BurstGPT reversal did not reproduce on physical hardware, while the Azure/Bailian-Qwen stable control did retain its ordering. This is a selected-case validation (2 of 13 panel policies, 3 sources, one operating region) — it is evidence about simulator-to-hardware fidelity in this selected case, not a general estimate of hardware reversal prevalence. See the manuscript's Limitations section for exact scope.

License / redistribution

Code (GitHub repository) is MIT-licensed. This dataset — all LSSP-derived files above — is released under the same MIT terms. No raw third-party workload trace files are included. BurstGPT (CC-BY-4.0), Azure LLM Inference Trace 2024 (CC-BY), and Bailian/Qwen (Apache-2.0) are referenced by canonical source, release tag/commit, and independently-verified SHA-256 only; this project has consistently treated all three as read-only-referenced rather than duplicated. See LSSP_THIRD_PARTY_SOURCE_LICENSES.md in this dataset for full per-source detail, redistribution status, and acquisition instructions.

Citation

See CITATION.cff in the GitHub repository. The archived-artifact citation is available now via the Zenodo DOI, 10.5281/zenodo.22306798; the manuscript's own journal citation will be added once one exists (this dataset makes no claim of journal publication or acceptance).

Limitations

• —RQ6 evaluated 2 of 13 panel policies, 3 sources, one operating region, one hardware/software environment, one physical execution per (policy, source, window) cell.
• —See "Not included in this revision" above for the specific deferred files and the reason for each.
• —METRIC_DEFINITION_SENSITIVITY and SLO_DEFINITION_SENSITIVITY robustness families have no implementing artifact; disclosed as a gap, not populated with an invented result.
• —Bailian/Qwen license confidence is MEDIUM (single-source confirmation).