t2ance/atlas-30-openmathreasoning-genselect-training

30. Does Qwen3.5-9B learn to use candidates on OpenMathReasoning?

1. Question and links

Trained by reinforcement learning on rows of eight OpenMathReasoning GenSelect candidates with one to seven of them correct, does Qwen3.5-9B's accuracy with eight candidates on held-out rows of the same distribution rise above its untrained accuracy and above the majority vote of the eight?

• —Report source: .claude/skills/atlas-experimenting/experiments/rl-training/30-openmathreasoning-genselect-training/main.tex in t2ance/ATLAS.
• —Issue: t2ance/ATLAS#70, the trainer-owned submit predictor and the composable potential, which this report's training is the first consumer of.
• —W&B runs, all in pqin/atlas-grpo: the four-card-plus-three memory check attempts nrtc229i, pkn4gkpq and tzdiu97r (the last one passed), and the eight-card sync memory check nhwjntqg (passed).
• —Status, 2026-09-20: closed. Every run is finished or stopped; the last formal run r30-cost0.1-from-overfit-boundaries-lr1e-5 was stopped at step 106 on the user's word. By the run's own validation on 256 held-out problems the accuracy did not rise (0.660 at step 40, 0.605 at step 105); the pre-registered baseline never ran. Report 31 continues the question.

2. Directory tree

• —README.md: this front page.
• —data/omr/merged.jsonl (884 MB): one line per problem of the GenSelect split whose answer was extracted from the forum, the candidates merged across the problem's prompts and deduplicated by text. Written by python -m training.scripts.import_omr_genselect merge on the home host, 33,251 problems and 367,132 candidates.
• —data/omr/labels.jsonl (174 MB): one line per candidate, the judge's verdict on whether the candidate's boxed answer is equivalent to the expected answer. Written by the label step against four Qwen3.6-27B servers on the home host, 2026-09-14 04:39 to 13:22 UTC.
• —data/omr/questions.jsonl, train.parquet, val.parquet, partition.json, stats.json: the built rows, written by the build step from the labels.
• —data/omr-overfit-64/train.parquet, val.parquet (the same 64 rows), selection.csv: the overfitting control's fixed set, 64 rows of data/omr/train.parquet chosen from the main run's training records where the step's eight samples split 2 to 5 correct, at most one of eight length-capped, mean response under 12,000 tokens, 16 rows a band of 2, 3, 4 and 5 correct, seed 1; selection.csv lists each row's question id, its pass count and mean response length in those records.
• —data/omr-overfit-32/train.parquet, val.parquet (the same 32 rows), selection.csv, partition.json: the 32-problem overfitting set, 32 rows of data/omr-overfit-64/train.parquet whose eight samples under the initial adapter split between 1 and 7 correct in both first steps (r30-overfit-64-grpo step 1 and r30-overfit-64 step 1), 8 per band of the original selection, seed 1; selection.csv carries both step-1 counts.
• —logs/r30-overfit-32-predictor.attempt1.log to attempt3.log, artifacts/r30-overfit-32-predictor/train/step_1/ to step_20/, runs/r30-overfit-32-predictor*/: the full-scheme overfitting control on that set (actor, critic and submit predictor at rank 64, PPO with GAE, batch 32 by 8), three segments of one W&B run (the actor at 2e-4 to step 12, 1e-4 for one unsaved step, 5e-5 from step 12 to step 19), 19 trained steps; step 20's records are the draw of the step-19 policy cut at 254 of 256 by the stop. Its saves stay on the node's disk (global_step_5, 10, 15, 19).
• —logs/r30-overfit-32-predictor-cost0.1.attempt1.log to .attempt4.log, artifacts/r30-overfit-32-predictor-cost0.1/ (train/step_16/ onward, potential/step_N.json, train_figures/), runs/r30-overfit-32-predictor-cost0.1*/: the full-scheme overfitting control continued from r30-overfit-32-predictor's global_step_15 with the explore cost raised from 0.01 to 0.10 and nothing else changed, steps 16 to 35 over four attempts (each resumed from the latest save); the question it answers is whether the policy's explore count moves when the cost does. Its saves stay on the node's disk (checkpoints/r30-overfit-32-predictor-cost0.1/global_step_{15,20,25,30,35}); its global_step_35 is the start of every formal run from the control (r30-cost0.1-from-overfit, the padded copy, r30-cost0.1-from-overfit-boundaries).
• —logs/r30-cost0.1.attempt1.log, artifacts/r30-cost0.1/ (train/step_N/, potential/step_N.json, train_figures/), runs/r30-cost0.1*/, checkpoints/r30-cost0.1/val_dumps/: the formal run on the full set (data/omr) from the three initial adapters under the overfitting control's reward and rates (the explore cost 0.10, the format score 0.1, the actor at 5e-5 and the critic at 2e-4 for two epochs) with the main run's batch shape and schedule (64 problems a step, the critic warmup 32, a validation every five steps, 200 steps). Its saves stay on the node's disk.
• —logs/r30-cost0.1-from-overfit.attempt1.log, artifacts/r30-cost0.1-from-overfit/, runs/r30-cost0.1-from-overfit*/, checkpoints/r30-cost0.1-from-overfit/val_dumps/: the same formal run started from the overfitting control's global_step_35 (its actor, critic and predictor) with no critic warmup, steps 36 onward; r30-cost0.1 stopped at step 5 in its favour (its five steps are the fixed initial actor's baseline on the full set, the validation 0.480). Its saves stay on the node's disk.
• —logs/r30-cost0.1-from-overfit-padded-smoke.attempt1.log, artifacts/r30-cost0.1-from-overfit-padded-smoke/, runs/r30-cost0.1-from-overfit-padded-smoke*/, checkpoints/r30-cost0.1-from-overfit-padded-smoke/ (the start global_step_35, the control's, without data.pt): the one-step smoke of the padded layout (32 problems by 8, packing off on the actor, the critic and the predictor, the budgets 81,920 / 196,608 / 81,920, no validation), 2026-09-18 19:47 UTC, launcher training/cost0.1_from_overfit_padded_smoke_8sync.sh; it proves the predictor's k = 0 logits agree across a prompt's rows and reads the memory peaks.
• —logs/r30-cost0.1-from-overfit-padded.attempt*.log, artifacts/r30-cost0.1-from-overfit-padded/, runs/r30-cost0.1-from-overfit-padded*/, checkpoints/r30-cost0.1-from-overfit-padded/ (the start global_step_35, the control's, without data.pt; the anchors every five steps, the latest, val_dumps/), data/gpqa/val.parquet (the 198 GPQA diamond questions of report 25 rebuilt under this run's prompt by training.rl.prepare_data --rebuild-from --like, rule-graded, the second validation set): the formal run from the control's step 35 with the padding-free packing OFF (verl's packed forward gives Qwen3.5's linear-attention layers no sequence boundaries; see experimenting/20260918T084000Z/15.md), everything else as r30-cost0.1-from-overfit, so the two runs from the same start measure the packing's cost; launcher training/cost0.1_from_overfit_padded_8sync.sh, started when the smoke passed (2026-09-18, the 20:0x UTC round). NOT LAUNCHED: its smoke proved the padded layout correct (the prompt logit's within-question std 0.0000, five attempts: two shape defects of the padded path, a stale file, two out-of-memories at 81,920 and 61,440) and about twice as slow in the updates; the run was set aside for the boundaries run on the user's word of 22:1x UTC 2026-09-18; its checkpoints/r30-cost0.1-from-overfit-padded/globalstep35 stays as a copy of the control's step 35.
• —logs/r30-cost0.1-from-overfit-boundaries-smoke.attempt1.log, artifacts/r30-cost0.1-from-overfit-boundaries-smoke/ — the ONE-STEP smoke (32 problems, step 36 from the control's globalstep35, packed, the packed run's budgets) of the value models given the packed rows' boundaries (training/rl/trainer.py tokenforwardwithboundaries, fe2d9e9); sent 22:20:11 UTC 2026-09-18; proven at 22:34: the prompt logit's within-question std 0.0000 on all 32 questions, the per-question prompt logit and the critic's first-token value equal to the padded smoke's within 0.001. Its globalstep36 deleted (waste). Its artifacts and W&B runs were deleted on the user's word of 03:0x UTC 2026-09-19 (the log stays); the second attempt (`logs/r30-cost0.1-from-overfit-boundaries-smoke.attempt2.log`, `artifacts/r30-cost0.1-from-overfit-boundaries-smoke/`, 00:20:42 UTC 2026-09-19, b2002f1) added the actor's proof: its first mini-batch pgclipfrac 0 and ppokl 0 exactly. Its globalstep_36 deleted (waste).
• —logs/r30-cost0.1-from-overfit-boundaries.attempt*.log, artifacts/r30-cost0.1-from-overfit-boundaries/, runs/r30-cost0.1-from-overfit-boundaries*/, checkpoints/r30-cost0.1-from-overfit-boundaries/ — the formal run from the control's globalstep35 with the value models given the packed rows' boundaries (launcher training/cost0.1fromoverfitboundaries8sync.sh, differing from cost0.1fromoverfit8sync.sh only in its name; the fix in training/rl/trainer.py, fe2d9e9); the one-variable comparison against r30-cost0.1-from-overfit (the leak) from the same start; OMR and GPQA validation every five steps. Attempt 1 (22:46:39 UTC 2026-09-18 to 00:19 UTC 2026-09-19, steps 36 to 39) trained an actor without the packed rows' boundaries (the trainer's row-shape wrapper of 418bcd3 hid the name cuseqlens from verl's signature check; its first actor mini-batch clipped 2.6 to 4.7 per cent of its tokens) and was deleted on the user's word of 03:0x UTC 2026-09-19 (its saved step, artifacts, outputs and W&B runs); only its log logs/r30-cost0.1-from-overfit-boundaries.attempt1.log stays; attempt 2 (00:47:38 UTC 2026-09-19, b2002f1, the wrapper naming the boundaries) is the run, from the same globalstep35. Second segment (attempt3.log, experimenting/20260919T072700Z/): resumed from global_step_50 at 07:27:25 UTC 2026-09-19 with the actor at 2e-4 in place of 5e-5 and nothing else (launcher training/cost0.1_from_overfit_boundaries_actor2e-4_8sync.sh), on the user's word of 07:2x (eight hours of autonomous tuning, the training verdict to rise); the first segment's steps 36 to 50 held both boundary proofs at every step and read within noise of r30-cost0.1-from-overfit. The second segment ran steps 51 to 53 and was stopped by PID at 08:46:51 UTC 2026-09-19 (global_step_53 its latest save): under the cost 0.10 the policy at 2e-4 moved to one explore a trajectory (explores 1.76, 2.08, 1.02; zero-explore submissions 13, 28, 107 of 512) with the verdict flat (0.680, 0.646, 0.664) and the objective 0.595; artifacts/.../train/step_51/ holds two draws of the step-50 policy (1,024 records, told apart by attempt_id: the first segment's driver drew its step 51 before its stop).
• —logs/r30-cost0.02-from-boundaries50-actor2e-4.attempt*.log, artifacts/r30-cost0.02-from-boundaries50-actor2e-4/, runs/r30-cost0.02-from-boundaries50-actor2e-4*/, checkpoints/r30-cost0.02-from-boundaries50-actor2e-4/ (the start global_step_50, the boundaries run's copied with data.pt; the anchors every five steps, the latest, val_dumps/) — the formal run with the explore cost 0.02 (launcher training/cost0.02_from_boundaries50_actor2e-4_8sync.sh: the boundaries run's global_step_50, the actor at 2e-4, cost.per_explore 0.02 in place of 0.10, nothing else), sent 08:47:15 UTC 2026-09-19 under the user's word of 07:2x (eight hours of autonomous tuning, the training verdict to rise), steps 51 onward; attempt experimenting/20260919T084700Z/. Closed 11:08 UTC 2026-09-19 at its anchor global_step_50 (the saves 55 and 57 deleted as broken policies): steps 51 to 54 read 0.678, 0.623, 0.721, 0.756, then the actor's updates broke the 0.1 line (step 54 ppokl -0.10/-0.12/-0.09, step 55 0.40/1.50/0.34) and the policy after them made no call at all (`attempt2.log`, the actor at 1e-4 from `globalstep_55`, steps 56 and 57: 0 explores, 0 submits, the entropy 3.4).
• —logs/r30-cost0.02-from-boundaries50-actor1e-4.attempt*.log, artifacts/r30-cost0.02-from-boundaries50-actor1e-4/, runs/r30-cost0.02-from-boundaries50-actor1e-4*/, checkpoints/r30-cost0.02-from-boundaries50-actor1e-4/ (the start global_step_50, the boundaries run's copied with data.pt; the anchors every five steps, the latest, val_dumps/) — the formal run with the explore cost 0.02 and the actor at 1e-4 (launcher training/cost0.02_from_boundaries50_actor1e-4_8sync.sh: the cost-0.02 run's launcher with the NAME and the rate 1e-4 in place of 2e-4, nothing else), sent 11:08:51 UTC 2026-09-19 under the user's word of 07:2x, steps 51 onward; attempt experimenting/20260919T110800Z/. Closed 16:08:53 UTC 2026-09-19 by the driver's PID on the user's word for the rerun from step 35, during step 63's generation, at its latest save global_step_62 (the anchors 50, 55, 60 kept): steps 51 to 62 read 0.689, 0.609, 0.738, 0.746, 0.699, 0.650, 0.684, 0.738, 0.656, 0.742, 0.635, 0.801; the validations OMR 0.648 / 0.629 and GPQA 0.803 / 0.828 at steps 55 and 60; step 61's actor updates 0.106 / 0.479 / 0.154 over the 0.1 line; the critic's explained variance within 0.02 of zero and the predictor's Brier 0.17 to 0.25 throughout. The verl run ulseukgb.
• —logs/r30-cost0.1-from-overfit-boundaries-lr1e-5.attempt*.log, artifacts/r30-cost0.1-from-overfit-boundaries-lr1e-5/, runs/r30-cost0.1-from-overfit-boundaries-lr1e-5*/, checkpoints/r30-cost0.1-from-overfit-boundaries-lr1e-5/ (the start global_step_35, the boundaries run's copied without data.pt; the anchors every five steps, the latest, val_dumps/) — the formal run with the three models at 1e-5 (launcher training/cost0.1_from_overfit_boundaries_lr1e-5_8sync.sh: the boundaries run's launcher with the NAME, the actor 5e-5 -> 1e-5, the critic 2e-4 -> 1e-5 and the submit predictor 6e-5 -> 1e-5, nothing else; the cost 0.10), sent 16:12:52 UTC 2026-09-19 (the launcher 2467262, the driver 2467268) on the user's word of 16:0x (all steps back to the step-35 checkpoint, the three rates to 1e-5, nothing else moved, a new W&B run under a new name), steps 36 onward; attempt experimenting/20260919T161500Z/; the verl run d6ndb5u4, its id fixed on the launch command. Second segment (attempt2.log, experimenting/20260919T231500Z/): resumed from global_step_51 after its save (sent 23:09:50 UTC 2026-09-19, the launcher 2972134, the driver 2972140) with the actor's fused output head on verl's torch backend in place of triton (actor_rollout_ref.model.fused_kernel_options.impl_backend=torch, launcher training/cost0.1_from_overfit_boundaries_lr1e-5_fusedtorch_8sync.sh, nothing else) on the user's word of 22:5x, the same W&B run continued; the rule: one step, its update_actor seconds per million tokens per epoch against the first segment's 66 to 68; clearly under continues, inside or a failure stops. artifacts/.../train/step_52/ holds two draws of the step-51 policy (1,024 records, told apart by attempt_id): the first segment's driver had completed its draw of step 52 and was in the updates when stopped, and the second segment redrew the step. Step 52 on the torch backend read 42.9 s per million tokens per epoch for the actor's update (258 s against 404 to 511 s a step before, MFU 0.53 against 0.34; the critic's and the predictor's updates unchanged; the first mini-batch at pgclipfrac=0 ppokl=0), so the run continues on the torch backend from step 52 onward.
• —logs/label.log, logs/judge_0.log to judge_3.log: the labelling client and its four judge servers on the home host.
• —logs/judge_node.log: the judge served on the eight-card node's card 7 during the four-plus-three memory check. logs/judge_home_for_node.log and logs/judge_tunnel.log: the judge served at home for the eight-card sync check, reaching the node through a reverse SSH tunnel.
• —logs/r30-genselect-9b-4train-3rollout-memory.attempt1.log to attempt4.log: the four-plus-three memory check's four attempts.
• —logs/gpu_watch/: the home host's card watcher during the labelling.
• —artifacts/r30-genselect-9b-4train-3rollout-memory/: that check's memory_check.json, its run.json manifest and its figures.

3. How to read each kind of file

• —Text, JSON and log files at <repo>/raw/main/<path>, for example raw/main/data/omr/stats.json and raw/main/logs/label.log.
• —.parquet and the large .jsonl at <repo>/resolve/main/<path>, for example resolve/main/data/omr/train.parquet; raw/ returns only the LFS pointer for those.
• —merged.jsonl and labels.jsonl are JSON lines: one object per line, read them a line at a time rather than loading the file.
• —train.parquet and val.parquet read with pandas.read_parquet or pyarrow.parquet.read_table; extra_info and reward_model are nested objects, so to_pylist() is easier to read than a dataframe.
• —artifacts/r30-genselect-9b-8sync-predictor.tar, artifacts/r30-overfit-64.tar — two closed runs' artifact directories packed as one tar each on 2026-09-19 (34,035 and 14,125 files, 6.2 and 3.0 GB) because the data root had passed the Hub's 100,000-file limit (publish 117 refused 100,655 files); the tar holds the directory as it was, unpack with tar -xf artifacts/<run>.tar -C artifacts/; the tars are on the Hub only (the GitHub reading copy leaves them out, as it does saved steps).
• —artifacts/r30-cost0.1-from-overfit.tar, artifacts/r30-cost0.1-from-overfit-boundaries.tar, artifacts/r30-cost0.02-from-boundaries50-actor1e-4.tar, artifacts/r30-cost0.02-from-boundaries50-actor2e-4.tar — four more closed runs' artifact directories packed the same way on 2026-09-20 at the report's close (16,934, 11,864, 7,727 and 4,134 files), for the same file limit; unpack with tar -xf artifacts/<run>.tar -C artifacts/; on the Hub only. The last formal run r30-cost0.1-from-overfit-boundaries-lr1e-5, which report 31's diagnosis reads, and the overfitting controls stay as directories in both places.
• —No adapter is published yet. When one is, it comes down with snapshot_download(repo_id, repo_type="dataset", allow_patterns="checkpoints/<dir>/global_step_N/actor/lora_adapter/*") and loads with PeftModel.from_pretrained(<base>, <local path>).
• —No W&B export is here yet: no training run has produced one. The memory checks' own readings are in artifacts/<run>/memory_check.json, whose real_step.timing_s holds the per-stage seconds and whose steps holds the synthetic steps' peaks.

4. Row fields

merged.jsonl, one object a line: id (the problem, omr- and six digits), problem, answer (the expected answer as extracted from the forum), pass_rate_72b_tir, problem_source, candidates (a list in first-appearance order, each text and boxed, where boxed is the content of the candidate's last \boxed{}).

labels.jsonl, one object a line: id and index name a candidate by its position in that problem's candidates list; correct is the verdict; judged is false when the judge gave none; reasoning is the judge's sentence, or the reason there is no verdict; finish_reason is stop, or length when the reply was cut at both token caps.

train.parquet and val.parquet, one row a problem, six columns: data_source (atts_omr), agent_name (atlas_sequential), ability (orchestration), prompt (the system and user messages), reward_model (style, and ground_truth, the expected answer), and extra_info. extra_info holds question_id, question, benchmark (omr), benchmark_spec (the name and the judge the reward manager asks), max_explores (8), correct_of_eight (how many of the row's eight candidates are correct), pass_rate_72b_tir, and cached_candidates, the eight candidates in the order the model reveals them, each with answer, reasoning, cost_usd, timed_out and is_correct.

partition.json: population_ids, every kept problem in id order, and train_ids, the first 80 per cent of them. The held-out problems are the rest; val.parquet carries only the first 256 of those, which is what the trainer validates online, so the whole held-out set is read from partition.json and not from val.parquet.

stats.json: problems_merged, problems_with_eight_candidates, problems_kept, train_rows, held_out_rows, val_rows_online, correct_of_eight (the problems at each count from one to seven), candidates_judged, candidates_correct, candidates_unjudged_dropped, seed, cost_usd_per_candidate, and judge (the model, its address, thinking off, and the sampling).

artifacts/<run>/memory_check.json: steps holds each synthetic step's stages with their seconds and per-rank peaks; real_step holds one real step's timing_s, step_seconds, trajectories, metrics, and the wake trace and margins where the layout produces them; token_budgets, production_batch and ppo_epochs record the shape it measured.

5. Weights and inputs

No weight has been shared from this directory: nothing is trained yet, and a weight moves only when the user names it. When one is shared it appears under checkpoints/ and is listed here.

What GitHub's reading copy leaves out, and the Hub repository keeps: the per-token arrays of any trajectory record (probabilities, token ids, masks), tensor archives, tokenizer.json, and the two data files over GitHub's 100 MB limit, data/omr/merged.jsonl (884 MB) and data/omr/labels.jsonl (174 MB), which the user chose to publish on the Hub alone (2026-09-15). Everything else is the same file on both sides.

Inputs, none of them copied into this directory:

• —OpenMathReasoning, nvidia/OpenMathReasoning, the fourteen genselect-*.parquet shards, downloaded 2026-09-13 to Experiment/datasets/hub/nvidia__OpenMathReasoning/data/ on the home host.
• —The model trained: Qwen3.5-9B, Hub snapshot c202236.
• —The judge: Qwen3.6-27B, Hub snapshot 6a9e13b, served in non-thinking mode at temperature 0 with the NeMo-Skills judge prompt carried verbatim in Experiment/core_code/benchmarks/omr.py.