t2ance/atlas-18-full-benchmark-closing-selector-transfer

ATLAS report 18: does the comparison closing carry ATLAS to the neutral selector on full GPQA and LiveCodeBench?

Complete raw products of ATLAS rl-training report 18 (GitHub issue #39). Three selector surfaces over every question of the canonical LiveCodeBench (175) and GPQA (198) validation sets, each question with all eight of its cached candidates revealed: the current ATLAS forced-submit surface, the same surface with only its closing user message replaced by a candidate-comparison instruction, and a neutral selector shown the same eight candidates with no ATLAS prompt and no tool history. One Qwen3-14B, one greedy deliberative rollout per question per surface, one TP2 vLLM process, 1119 generations.

Start here

report/report18.pdf, the whole write-up. Everything else here is the evidence it cites.

The headline, in short

There is no neutral-to-ATLAS gap to recover at benchmark scale. The neutral selector reads 0.806 on LiveCodeBench against current ATLAS's 0.789 (paired +0.017 [-0.017, +0.051]) and 0.859 on GPQA against 0.848 (+0.010 [-0.015, +0.040]). Report 15's 0.179 gap was a property of a 39-pair probe, not of the benchmarks.

The closing-only surface lands on the neutral value on LiveCodeBench and does not move on GPQA. LiveCodeBench 0.806 (delta vs current +0.017 [-0.011, +0.046], 5 questions gained, 2 lost); GPQA 0.843 (-0.005 [-0.025, +0.010]). On the 39 and 38 mixed questions where a selector decision exists: +0.077 [-0.051, +0.205] and -0.026 [-0.079, 0.000]. Every correctness interval covers zero.

The closing controls reasoning length and raises the correct path mass on LiveCodeBench, without changing decisions. Median think 837 to 2244 tokens (paired +1427 [1090, 1774]); correct path mass 0.754 to 0.811 (+0.056 [0.021, 0.093]); share choosing the first candidate 0.263 to 0.389 (+0.126 [0.063, 0.194]).

Composition of the candidate sets (why the full set is saturated): LiveCodeBench 107 of 175 questions have all eight candidates correct, 26 have none, 9 have every slot timed out; GPQA 145 of 198 all correct, 14 none, and 155 candidate sets carry one answer letter.

The three surfaces

File tree

README.md                                   this file
FILES.txt                                   index of every file
report/
  report18.pdf                              THE REPORT, read this first
  report18.tex                              its LaTeX source
manifest.json                               the 373 questions, every candidate's render hash and label, checkpoint file hashes
audits/
  full_set_surfaces.json                    the preflight: surface texts and hashes, every state's conversation hash, prompt token counts
<surface>/qwen3_14b_base/
  <experiment>.jsonl                        373 rows: think text, submit lead, decision tokens, branch points, full conversation
  <experiment>.parquet                      the same rows
  <experiment>.meta.json                    that run's effective configuration; the neutral run's `prompt_arm`,
                                            `closing_sha256` and `closing_text` fields carry the runner's unused
                                            `--prompt-arm` default (the neutral surface has no ATLAS arm), the rows
                                            carry the surface's own arm `neutral` and closing 264f6e02...
trace_packets/
  <surface>.jsonl                           373 deterministic packets per surface
  INDEX.json                                the packet index
tables/
  headline.csv                              per (benchmark, subset, surface) metrics
  paired_delta.csv                          paired differences: closing, residual, gap
  matrix.csv                                the 2x2 correctness counts
  positions.csv                             selected-position histograms
  composition.csv                           subset sizes
  states.csv                                the per-question join of the three surfaces
  rows.csv                                  every scored row
  trace_index.csv                           the trace cohorts
  trace_reading_lcb.md, _gpqa.md            the reading of the 21 cohort questions, three surfaces each
smoke/                                      the smoke test's rows (2 LCB + 2 GPQA per surface)
logs/
  serve_qwen3_14b_base_tp2.log              the vLLM server log
  run_<surface>.log                         one log per run

How to read a file

A text file is served at https://huggingface.co/datasets/t2ance/atlas-18-full-benchmark-closing-selector-transfer/raw/main/<path>.

A .parquet is stored through LFS, so raw/ returns only a small pointer. Use resolve/main/<path> instead. FILES.txt is the complete index; do not rely on tree/main?recursive=true, which pages.

Trace packets and the recorded reading

trace_packets/<surface>.jsonl places, per question: the correct positions and the answer-bearing positions, the selected position and whether it was correct, the path mass (ATLAS) or first-token distribution (neutral), the full think text, the submit-lead text, the selected answer, and the first branch point where several codes were still reachable. Nothing in them classifies the reasoning.

tables/trace_reading_lcb.md and tables/trace_reading_gpqa.md record a reading of 21 questions, three surfaces each: every closing rescue and closing break (LCB 7, GPQA 3), the whole neutral-and-closing-against- current cohort (8, all among the former), and 16 hard mixed questions (LCB 12, GPQA 4 further). Each file opens with a summary table, then one section per question: what each surface's reasoning did, with quoted excerpts, whether the comparison closing produced a concrete check the current surface did not, whether that check contained a wrong step, and whether the submission followed the reasoning. The classification of the closing relative to the current prompt uses verification_helped, verification_erred, preference_only, mismatch, unchanged, and runaway_lead; the totals are 3, 0, 5, 1, 10, and 2.

Provenance

• —Model: Qwen/Qwen3-14B, snapshot 40c069824f4251a91eefaf281ebe4c544efd3e18, untrained base weights, bf16.
• —Serving: vLLM, --tensor-parallel-size 2 over 2x A100 80GB, max_model_len 32768, prefix caching on, 743,840-token KV cache, 54.7 percent prefix-cache hit rate.
• —Decoding: greedy, temperature 0. Think budget 8192 tokens, submit-lead and decision budgets 6144 tokens.
• —All three runs shared the one server process; 48 workers each; 08:37 to 09:48 UTC on 2026-09-04.
• —Code: t2ance/ATLAS, branch feature/full-set-closing-transfer, Experiment/core_code/scripts/decision_value/.
• —Companion datasets: t2ance/atlas-17-prompt-interpolation-selector-recovery (report 17), t2ance/atlas-16-verifier-permission-prompt-ablation (report 16), t2ance/atlas-15-selector-capacity-vs-training (report 15).