kobzaond/RLVRAMBench

1---2pretty_name: RLVRAMBench — measured memory feasibility and configuration decisions3license: mit4language:5- en6tags:7- benchmark8- reinforcement-learning9- gpu-memory10- reproducibility11- resource-configuration12task_categories:13- other14configs:15- config_name: admission_candidates16  default: true17  data_files:18  - split: test19    path: benchmark/decision/results/candidate_results.csv20- config_name: transfer_configurations21  data_files:22  - split: measured23    path: benchmark/transfer_configurations.csv24- config_name: all_configurations25  data_files:26  - split: measured27    path: benchmark/configuration_results.csv28- config_name: eligible_runs29  data_files:30  - split: measured31    path: benchmark/runs.csv32- config_name: recorded_attempts33  data_files:34  - split: recorded35    path: benchmark/attempts.csv36- config_name: stage_measurements37  data_files:38  - split: measured39    path: benchmark/stage_measurements.csv40- config_name: transfer_queries41  data_files:42  - split: evaluation43    path: benchmark/queries.csv44- config_name: admission_attempts45  data_files:46  - split: recorded47    path: benchmark/decision/results/attempts.csv48- config_name: larger_model_configurations49  data_files:50  - split: measured51    path: benchmark/estimation/results/configurations_flat.csv52- config_name: expanded_host_configurations53  data_files:54  - split: measured55    path: benchmark/host_capacity/results/configurations_flat.csv56- config_name: configuration_feasibility57  data_files:58  - split: configurations59    path: tables/three-model-configurations.csv60---61 62# RLVRAMBench63 64**Which language-model training configurations can I use with the memory65I have, and how much testing does that decision require?**66 67RLVRAMBench is a measurement dataset with open evaluation tasks for a68specific language-model training system. It measures memory feasibility69when response generation and reinforcement-learning updates share the70same graphics processors. It provides measured outcomes, fixed prediction71tasks, a budgeted decision replay, reference methods, and raw failure evidence.72It does **not** score a model's intelligence or learning quality.73The companion article is **Memory Feasibility in Colocated Language-Model74Reinforcement Learning: A Failure-Aware Measurement Study**. This is a75focused, extensible study, not a comprehensive cross-platform benchmark76or evidence of community adoption.77 78The default table above is a readable twelve-candidate experiment. Each79row shows the model, workload, hardware, training settings, earlier source80label, new screening outcome, and a repeated evaluation outcome obtained81from **different random seeds**. The other views expose the broader82measurement corpus and transfer tasks.83 84Authors: Ondřej Kobza and Jan Šedivý, Czech Institute of Informatics,85Robotics and Cybernetics, Czech Technical University in Prague.86ORCID: [Kobza](https://orcid.org/0000-0002-0529-9860),87[Šedivý](https://orcid.org/0000-0003-0626-2303).88 89- [Paper PDF](paper/RLVRAMBench_IEEE_Access_submission.pdf)90- [Supplement](paper/RLVRAMBench_supplement.pdf)91- [Public reproduction code](https://github.com/kobzaond/RLVRAMBench-reproduce)92- [Benchmark instructions](BENCHMARK.md)93- [Data dictionary](DATA_DICTIONARY.md)94- [Pinned release and archive](release.json)95 96## What is measured?97 98Each process either completes its requested training and scheduled99operations, has a diagnosed memory failure, or remains unresolved for100resource-label purposes. External device-memory samples measure the101maximum used memory on **any one allocated device**, not a sum across GPUs.102 103For the three distinct eligible evaluation seeds of a configuration:104 105| Outcome | Meaning |106|---|---|107| `within_margin` | All three complete and each peak is at most 38,912 MiB |108| `above_margin` | All three complete, but at least one exceeds that limit |109| `memory_failure` | At least one eligible run has a diagnosed memory failure |110 111The limit leaves 5% below the measured capacity of 40,960 MiB per device.112It is a stated operating convention, **not a statistical guarantee**.113One mebibyte (MiB) is 1,048,576 bytes. Failed runs do not have a114counterfactual completed-run peak; blank completed-peak fields are not zero.115All three slots must have eligible outcomes before the repeated label116is assigned, even if a partial record already contains a diagnosed117failure. Unresolved attempts and known partial failure evidence remain visible.118 119These are conditional memory outcomes. They do not say that every120scheduler job, software invocation, or future repetition will complete.121The full attempt history is therefore separate from eligible labels.122 123## Ways to use the measurements124 125### 1. Predict a target outcome from existing measurements126 127A method receives explicit source measurements and target settings, but128not target outcomes. It predicts one of the three labels. For example,129can a configuration measured on four GPUs run on two?130 131The static suite contains 52 tasks and 400 queries across model, workload,132GPU-count, and run-duration transfer. Queries reuse **94 distinct target133configurations**. They are not 400 independent experiments.134Source-copy, always-approve and always-reject results are provided.135 136Use the public code repository:137 138```bash139python3 benchmark.py verify140python3 benchmark.py baseline --rule source_label --output predictions.csv141python3 benchmark.py evaluate --predictions predictions.csv --output scores.json142```143 144For another method, export allowed evidence with `benchmark.py inputs`,145then provide a CSV with `query_id,predicted_state`. The evaluator rejects146missing, duplicate, extra or invalid predictions. Reading the public147target tables while producing predictions violates this task contract.148 149### 2. Decide which candidates to admit as tests become available150 151The admission panel asks whether available source evidence helps identify152usable target configurations before all of them have been tested.153It contains four model–workload cases, three candidate settings per case,154and 48 fresh processes:155 156- One five-step screen per candidate can be requested by a rule.157- Three separate five-step evaluation seeds determine the target label.158- Screening outcomes never contribute to that evaluation label.159 160Source copy initially approves the donor's within-margin settings.161A headroom guard also requires an additional source-memory allowance.162Direct screening starts without approving any candidate and uses no donors.163All request screens in the same frozen order. Each invocation is charged164before its outcome is revealed; unrequested target results remain hidden165from the replay.166 167```bash168python3 decision_benchmark.py replay \169  --protocol benchmark/decision/protocol.json \170  --attempts benchmark/decision/results/attempts.json \171  --output admission-scores.json172```173 174`scores.json` includes each case, each budget, requested attempts, revealed175screens and resulting decisions. `summary.csv` is the compact comparison.176The protocol and prelaunch amendment are retained with their timestamps.177No memory failure is retried to obtain a favorable outcome. Unresolved178evaluation slots retain admissible-outcome bounds.179 180### Observed admission results181 182All 48 planned processes supply validated memory outcomes, with no183retries or unresolved slots. The twelve evaluation candidates contain184six within-margin configurations, two above-margin configurations, and185four memory failures. Screens and separate evaluation repetitions agree186in this panel.187 188| Rule | Case-averaged recall before testing | First tested common per-case budget with full recall | Prior donor attempts across four cases | Memory-failure / above-margin approvals |189|---|---:|---:|---:|---:|190| Copy the source label | 100% | 0 | 36 | 0 / 0 |191| Additional headroom guard | 75% | 1 | 36 | 0 / 0 |192| Direct screening | 0% | 3 | 0 | 0 / 0 |193 194Direct screening reaches 25%, 75%, and 100% case-averaged recall at195budgets of one, two, and three attempts per case. Case averaging gives196each model–workload combination equal weight, rather than pooling197candidates. The guard initially rejects two usable Qwen candidates.198Source copy and the guard differ only at those candidates, which are199queried first; their positive-budget decisions are **identical by200construction**, not two independent confirmations.201 202This is evidence that the specified existing measurements were sufficient203for this selected workload change, not a universal transfer result.204The separate GPU-transfer task below contains harmful approvals.205The first common budget attaining full recall is not a proof of the206minimum number of tests an optimized strategy would require.207Without an existing donor profile, the attributed counts for full recovery208are 36, 40, and 12 attempts respectively, with different work per attempt.209Thus the comparison does not establish cold-start or GPU-time superiority210of transfer. All three rules have 100% observed precision wherever they211approve anything; the small panel does not estimate future failure risk.212 213### 3. Compare estimators with complete model families held out214 215The separate `benchmark/estimation/` comparison asks whether architecture216and configuration information improves on copying an earlier outcome.217It uses ninety historical configurations. Each of three tests withholds218all thirty configurations of one model family and fits on the other sixty.219Seeds and alternative GPU counts of the test family are not fitting examples.220 221References include donor copying, empirical component regression with a222one-sided startup check, and regularized logistic classification.223The regression uses completed peaks without inventing numerical targets224for failures; the classifier also learns from failure labels.225These are not exact tensor-liveness models or calibrated failure probabilities.226 227Results differ by family: regression gets every Qwen label right but228approves four memory-failing Granite configurations; the classifier rejects229six usable Qwen settings. Read the separate approval errors, not just230accuracy. The historical labels were public during design, so this is231not blind method development.232 233The [estimator guide](benchmark/estimation/README.md) explains inputs,234frozen predictions, acquisition costs and CPU reconstruction.235 236### 4. Inspect prospective larger-model transfer237 238The `larger_model_configurations` view contains twelve planned Qwen2.5-7B239settings with predictions frozen before target execution. Each row shows240how many of its three seed outcomes are eligible, any known partial failure241evidence, the resolved label when available, and each method's prediction.242This is size transfer within a familiar model family, not a new-family test.243 244The original panel has 23 eligible startup memory-admission failures and24513 unresolved slots, with no completed training steps. Only six repeated246labels resolve, all to failure. Donor copying wrongly approves those six;247the fitted references reject them, but an always-failure rule does too.248Their scores on this subset do not demonstrate recovery of usable settings.249In particular, the regression's three approvals have unresolved outcomes.250 251The `expanded_host_configurations` view is a separate follow-up to252host-memory exhaustion in the original panel. It retains the frozen253predictions, uses new seed slots, and increases the host-resource254allocation. Its six settings are not replacements for original targets.255All nine two-GPU invocations fail the startup memory check. The nine256four-GPU invocations exit successfully but lack the required final-step257record, leaving their three repeated labels unresolved. No validated258completion is available in this panel either. See the259[follow-up guide](benchmark/host_capacity/README.md) for the diagnostic260evidence and separate allocation costs.261 262## What score should I report?263 264Report **within-margin recall at a stated measurement budget**, together265with separate memory-failure and above-margin approvals.266 267- **Recall:** what fraction of the genuinely within-margin candidates did268  the rule approve?269- **Approval precision:** what fraction of its approvals were within margin?270- **Approval errors:** how many approvals actually fail for memory, and271  how many complete but exceed the chosen limit?272- **Effort:** how many target attempts were requested? Was source evidence273  already available or did it have to be collected?274 275A zero denominator is `null`, not a perfect score. Approving nothing276avoids approval errors but recovers nothing; approving everything recovers277every usable candidate but may also admit failures. Neither should win by278hiding the other side of that trade-off.279 280There is intentionally no unexplained scalar combining errors and cost.281Those trade-offs depend on the user's application. Label accuracy is282provided for diagnosis, not as a universal deployment ranking.283 284An exact example from GPU-count transfer shows why this matters:285 286| Reference rule | Correct labels / 36 queries | Within-margin targets approved | Memory failures approved | Above-margin targets approved |287|---|---:|---:|---:|---:|288| Copy the source label | 20 | 20 | 2 | 4 |289| Approve everything | 26 | 26 | 4 | 6 |290| Reject everything as a memory failure | 4 | 0 | 0 | 0 |291 292“Approve everything” has higher label accuracy than copying, but admits293twice as many memory failures. Pooled accuracy also overweights easy tasks:294216 of the 400 queries are workload transfers solved by lookup on this295grid. The four duration queries offer little discrimination.296Always retain the per-track and per-case results.297 298## What does this release contain?299 300The static tables contain 212 settings, 612 eligible processes, 683301recorded attempts and 3,434 observed stage measurements. These include302the original 588-process corpus and 24 completed batch-by-logging controls.303Sixteen historical source screens are counted separately in the archive.304The 48-slot admission panel is separate; it is not silently added to305the unchanged static-table totals.306The estimator comparison reuses ninety of those historical configurations;307its three test folds are not additional GPU experiments.308The original larger-model panel and expanded-host follow-up have their own309configuration and process tables. Their planned slots, unresolved outcomes,310and costs are not included in the static or admission totals above.311 312The default candidate table is a useful entry point. `configurations.csv`313contains settings without outcomes. `configuration_results.csv` and314`transfer_configurations.csv` join them for browsing. `runs.csv`,315`attempts.csv`, and `stage_measurements.csv` preserve process, invocation,316and execution-stage detail. They have different units and must not be317concatenated into one dataset of independent examples.318 319The compressed review archive contains raw training logs, external320memory traces, allocator records, environment provenance, frozen matrices,321attempt exclusions, and reconstruction code. Model weights and optimizer322checkpoint payloads are **not** redistributed.323 324## Cost accounting325 326For static prediction, source-cost fields count **supplied evidence**;327they are not measured acquisition effort by a method.328 329For admission replay, three ledgers remain separate:330 3311. The donor evidence already available to transfer-informed rules.3322. The target attempts actually requested at each budget.3333. All physical runs collected to evaluate the rules, including hidden334   evaluation seeds.335 336Cold-start accounting charges the donor investment as well. It does not337make those measurements free or establish superiority over a direct338strategy allowed an equally large budget of extra target repetitions.339One donor step and five target steps also have different runtime costs.340Recorded GPU-seconds exclude queue time and other allocation overhead.341 342## Scope and appropriate reuse343 344The experiments use compact language models with low-rank adaptation and345Group Relative Policy Optimization, in a fixed VERL/vLLM execution stack.346Generation and training share NVIDIA A100 devices. They ran on347[Karolina](https://www.it4i.cz/en/infrastructure/karolina), a supercomputer348at IT4Innovations in Ostrava, Czech Republic.349 350Use the records to evaluate resource-label prediction, compare measured351admission decisions, audit failures, study why stage and overall peaks352differ, or reconstruct the paper. Do not use them to claim a general353failure probability, convergence benefit, safe full training schedule,354or transfer to unmeasured hardware and model scales.355 356The cases were selected, not randomly sampled. New admission outcomes357come from familiar model/workload families; their four cases are not35848 independent cases. The public labels make inspection possible but359also mean this is not a permanently hidden test set. Strong claims about360a new predictor need genuinely unseen cases beyond this release.361 362Some historical executions used modified source working trees without363captured per-run patches. Their commit IDs do not by themselves reconstruct364those modifications. Later controls and the admission panel use immutable365source copies. Rebuilding results from saved traces is distinct from366recreating every historical GPU execution.367 368## Reproduce and cite369 370The code repository pins an immutable evidence archive and verifies its371SHA-256 digest and extracted file manifest. CPU-only reconstruction372regenerates results from raw evidence before comparing them with the373published tables. Optional filesystem isolation hides those reference374tables and the original project during analysis.375 376GPU reruns are a separate, more demanding task: obtain upstream models377under their terms and adapt the recorded environment and cluster-specific378launchers. Keep all virtual environments inside their project directory.379 380Original benchmark measurements and reproduction software are MIT381licensed. Third-party data, model, software and template terms remain382separate; see [RIGHTS.md](RIGHTS.md). The archive is not a relicensing383of upstream material. Both benchmark and code repositories are public384and ungated. No DOI is claimed until the authors create one.385