t2ance/atlas-31-strengthening-candidate-verification-under-rl

31. Strengthening candidate verification under reinforcement learning

1. Question and links

Read this first. The reading copy of this directory is t2ance/atlas-experiments under 31-strengthening-candidate-verification-under-rl/; the saved training steps and the per-token training arrays are on the Hugging Face repository t2ance/atlas-31-strengthening-candidate-verification-under-rl only.

How can reinforcement learning make the orchestrator's comparing and verifying of candidates emerge or grow stronger, so that accuracy rises with the candidates read toward the best of them and the policy settles neither on one call nor on the vote?

• —Report source: .claude/skills/atlas-experimenting/experiments/rl-training/31-strengthening-candidate-verification-under-rl/main.tex in t2ance/ATLAS.
• —Issue: none.
• —W&B runs, in pqin/atlas-grpo: the training r31-techniques-grpo, started 2026-09-20 22:39 UTC (group r31-techniques-grpo: the trainer's run 1b5e1ae4, https://wandb.ai/pqin/atlas-grpo/runs/1b5e1ae4 , the trajectory metrics' run ddd8d28b, the actor updates' run 72de9a45); the smoke r31-techniques-grpo-smoke (633dee41, its trajectory metrics a7c87192, its actor updates fb18949f).
• —Status, 2026-09-20: the diagnosis of report 30's run and the zero-shot probes of the untrained model are finished; the rows of method 1 are built; the smoke ran two steps; the training r31-techniques-grpo is running (235 steps, a validation every 5).

2. Directory tree

• —README.md: this front page.
• —diagnosis/scripts/: the programs that produced the Background's numbers, run from diagnosis/ (they read report 30's data root by absolute path and write beside themselves): load.py, load2.py (one row a trajectory), a1.py to a3.py (the policy, the critic, the submit predictor), b1.py to b3.py (selection on disagreeing candidates), m1.py (wording and thinking length), render.py to render3.py (decoded traces).
• —diagnosis/scripts/r31_markers.py, r31_nll.py with diagnosis/r31_markers.parquet, r31_nll.parquet: one row a trajectory of this report's training draws (steps 1 to 20 at the cost 0.05): the turns' lengths and checking words, and the sampled text's negative log-probability by turn; the appendix's step 2 quotes them.
• —The suffix .cost0.05 under artifacts/r31-techniques-grpo/ (train/step_16 to step_21, val/step_15, val/step_20) and checkpoints/r31-techniques-grpo/val_dumps/ marks what the training's first attempt wrote after step 15 at the cost 0.05, before it diverged and was resumed from step 15 at 0.01; the same names without the suffix are the resume's. On the node the save of step 20 is kept as checkpoints/r31-techniques-grpo/cost0.05_diverged_step_20.
• —diagnosis/*.parquet, *.csv, *.json: their tables; labels180.parquet is the 180 labelled traces joined with traces3_index_blind.csv.
• —diagnosis/traces/ (16, stratified by outcome), traces2/ (32, three or more explores with disagreeing candidates), traces3/ and traces3b/ (180, random within a phase; 3b is the blinded copy the readers saw).
• —diagnosis/readings/: the three reader workflows' journals (one result line a reader) and their scripts.
• —data/omr-verification-techniques-cost-unstated/: report 30's OMR rows rebuilt for method 1 (import_omr_genselect.py build --cost-per-explore 0.10 --cost-unstated --verification-techniques, source 3ca3ea4): the verification techniques in the system message, the opening message without a cost section, the cost 0.10 an explore charged by the row; the same questions, order and candidates as report 30's data/omr. train.parquet (15,968 rows), val.parquet (256), questions.jsonl, partition.json, stats.json. The smoke r31-techniques-grpo-smoke read these rows.
• —data/omr-techniques-cost0.05-unstated/: the training's rows: the build at source 8c72332 (the verification techniques as a two-column table in plain words) with --cost-per-explore 0.05 --cost-unstated --verification-techniques; every opening message equals report 30's rows', extra_info.cost is {per_explore: 0.05, per_usd: 0.0, stated: false}. The training r31-techniques-grpo reads these rows.
• —data/omr-techniques-cost0.01-unstated/: the same build with --cost-per-explore 0.01 (the user's value of 2026-09-21); every prompt, problem order and candidate list equals the 0.05 rows', extra_info.cost is {per_explore: 0.01, per_usd: 0.0, stated: false}. The training reads these rows from its resume at step 15 (07:46 UTC 2026-09-21).
• —zero-shot-probe/omr-024738/<setting>/: one zero-shot trajectory of the untrained Qwen3.5-9B on validation question omr-024738 (candidates split 4 to 4 between 29520 and 141, the truth 141) for each of three settings: default (the unchanged system message, the cost stated), verification (the techniques, the cost stated), verification-nocost (the techniques, the opening message of report 30's run without a cost section). input.txt and output.txt are Qwen's chat template with no comment added, submission.json the submit call's arguments, meta.json the turns; zero-shot-probe/probe.py produced them against a temporary vLLM server (64K context, temperature 1.0, top-p 0.95, top-k 20, the training line's structural tag and observations).
• —zero-shot-probe/cost-statement/<setting>/<question>-s<seed>/: the untrained Qwen3.5-9B under the verification techniques with four opening messages: cost-0.1 (the current cost section), cost-0.01 (the same section and the observations' charge at 0.01), cost-0.1-no-last-line (without "You may submit without exploring."), no-cost (no cost section); 12 validation questions with 3 to 5 correct candidates by 4 samples, 64K context, run 2026-09-20. 177 of 192 finished; the 15 stopped after 49 minutes (7 of cost-0.01, 2 of cost-0.1, 6 of no-cost, on omr-024634, -024636, -024641, -024649) have an input-less folder and no meta.json. meta.json carries judged_correct from the project's OMR judge (grade_cost.py); summary.csv is one row a trajectory; probe_cost.py produced the trajectories.
• —logs/: the run logs and the machine layer's card samples (logs/gpu_watch_node/), written once a run starts; logs/judge_tunnel.log, the home host's tunnel to the judge.
• —artifacts/<run>/, checkpoints/<run>/, outputs/<run>/, runs/<run>/, wandb/: written by the runs of this report: r31-techniques-grpo-smoke and r31-techniques-grpo; artifacts/<run>/train/step_N/ and val/ hold one JSON a trajectory.

3. How to read each kind of file

• —Text, CSV, JSON and log files at <repo>/raw/main/<path>, for example raw/main/zero-shot-probe/cost-statement/summary.csv.
• —.parquet at <repo>/resolve/main/<path>, for example resolve/main/data/omr-verification-techniques-cost-unstated/val.parquet; raw/ returns only the LFS pointer for those. Read with pandas.read_parquet; extra_info and reward_model are nested objects.
• —zero-shot-probe/**/input.txt and output.txt are Qwen's chat template as plain text: the input is everything the model saw before its first token, the output everything after it, the observations included.
• —No adapter is published. When one is, it comes down with snapshot_download(repo_id, repo_type="dataset", allow_patterns="checkpoints/<dir>/global_step_N/actor/lora_adapter/*") and loads with PeftModel.from_pretrained(<base>, <local path>).
• —A W&B export is runs/<run>/<id>/: history.csv holds a row a step; the validation curve is val-core/atts_omr/acc/mean@1 against training/global_step.

4. Row fields

data/omr-verification-techniques-cost-unstated/train.parquet and val.parquet, one row a problem, the six columns of report 30's data/omr rows (data_source, agent_name, ability, prompt, reward_model, extra_info); prompt[0] is the system message with the verification techniques, prompt[1] the problem and the budget with no cost section; extra_info.cost is {per_explore: 0.1 or 0.05, per_usd: 0.0, stated: false} by directory; the other keys of extra_info are report 30's (question_id, question, benchmark, benchmark_spec, max_explores, correct_of_eight, pass_rate_72b_tir, cached_candidates with answer, reasoning, cost_usd, timed_out, is_correct). stats.json adds verification_techniques.

zero-shot-probe/**/meta.json: qid, setting (or mode), seed, explores, end (submit, no_call, context), turns (the tokens and the call of each turn), answer, truth, matches_candidate, correct_by_candidate_match, answer_among_revealed, candidates (answer and correctness in reveal order), judged_correct and judged (the project's OMR judge). zero-shot-probe/cost-statement/summary.csv: one row a trajectory: s setting, q question, k seed, N explores, end, t1 tokens of the first turn, tot tokens of all turns, ok the judged verdict, says_cost, c1 whether the first candidate is correct, cov whether a revealed candidate is correct, mixed whether the revealed candidates disagree in correctness.

diagnosis/labels180.parquet: one row a labelled trajectory of report 30's run with its phase and the readers' labels (the scripts under diagnosis/scripts/ name every column they write).

5. Weights and inputs

• —No weight is shared from this directory. The runs start from the untrained Qwen/Qwen3.5-9B (Hugging Face, snapshot c202236235762e1c871ad0ccb60c8ee5ba337b9a) with the rank-64 initial adapter r30-actor-init of report 30 (t2ance/atlas-30-openmathreasoning-genselect-training, checkpoints/r30-actor-init/).
• —The rows are built from report 30's data/omr/merged.jsonl and labels.jsonl (the same repository), which come from nvidia/OpenMathReasoning's GenSelect split judged by Qwen3.6-27B.
• —The diagnosis reads report 30's trajectory records artifacts/r30-cost0.1-from-overfit-boundaries-lr1e-5/ (the same repository).
• —GitHub alone leaves out the per-token arrays of trajectory records and any saved step.
• —zero-shot-probe/sketch-row/<question>-s<seed>/: the untrained Qwen3.5-9B under the techniques table with row 0 (the sketch before the first explore; table_v2.txt, the text that went into training/task.py at b2063d3), no cost shown, four questions by four seeds, 64K context, run 2026-09-20 (probe_v2.py). 11 of 16 finished; the 5 stopped after 15 minutes (omr-024641 seed 1, omr-024634 seeds 0 to 3) have no meta.json. First turns: median 5,460 tokens, 2 of 11 under 2,000, 6 of 11 never explored; three trajectories name "Technique 0" and count their whole solution as the sketch. The user kept the wording for the training.
• —zero-shot-probe/plain-two-column/<question>-s<seed>/: the untrained Qwen3.5-9B under the text the training uses (the two-column plain-words techniques, table_v4.txt, equal to training/task.py at 8c72332), no cost shown, omr-024738, omr-024620 and omr-024641 by seeds 0 to 3, 64K context, run 2026-09-20 (probe_v3.py); all twelve finished. Against the same questions and seeds under the numbered four-column table (sketch-row/): first turn median 2,072 tokens against 5,460, 6 of 12 under 2,000 against 2 of 11, 2 never explored against 6, explores 3.00 against 1.82; no trajectory ticks a technique off by name.