t2ance/atlas-17-prompt-interpolation-selector-recovery

ATLAS report 17: which ATLAS prompt component suppresses candidate discrimination?

Complete raw products of ATLAS rl-training report 17 (GitHub issue #38). Eight prompt arms interpolate from the frozen ATLAS orchestrator prompt to an empty system message with a neutral closing, plus the neutral A/B selector rerun on the same server as the interpolation anchor. One Qwen3-14B, 78 fixed states, greedy, one vLLM process.

Start here

report/report17.pdf, seven pages, the whole write-up. Everything else here is the evidence it cites.

The headline, in short

The gap this study was built to decompose did not reproduce. On one server in one session the ATLAS submit surface reads 0.551 and the neutral A/B surface reads 0.590 over the same 78 states, a paired difference of +0.038 with a 95% interval of [-0.077, +0.154]. Report 15 measured 0.705 against 0.526, a gap of 0.179.

The reason is in report 15's own data: its neutral deliberative rate was 0.705 under greedy and 0.603 under seed 101, a 10.3 pp spread between two rollouts of one configuration, while its submit-surface rate was 0.5256 under both. The neutral surface is the unstable term.

Consequence: gap recovery R = (A_arm - A_current) / (A_neutral - A_current) divides by 0.038, a denominator smaller than its own interval. The values in tables/arm_recovery.csv are kept for completeness, several exceed 1, and none of them carries information.

What the run does establish, in raw units:

• —The closing instruction controls reasoning length. Replacing only The explore budget is now closed. Submit your final answer now. with a statement of the comparison objective raises median think tokens from 2239 to 4820, a paired +1941 [1262, 2621] - more than replacing the entire system message (+1280) - and accounts for essentially the whole reasoning-length difference the neutral surface shows (+1835). These are the only intervals in the study that exclude zero.
• —No arm moves accuracy outside noise, an empty system message with a neutral closing included. Every accuracy interval covers zero.

The eight arms

verifier_permission (report 16's arm) is also registered in audits/prompt_arms.json and was not rerun.

File tree

README.md                              this file
FILES.txt                              index of every file
report/
  report17.pdf                         THE REPORT, read this first
  report17.tex                         its LaTeX source
manifest.json                          the 78 states; byte-identical to report 16's
audits/
  prompt_arms.json                     all nine arms' system and closing texts with sha256
<arm>/                                 one directory per arm
  qwen3_14b_base/submit_pair.jsonl     156 rows (78 states x 2 modes)
  qwen3_14b_base/submit_pair.parquet   the same rows as parquet
  qwen3_14b_base/submit_pair.meta.json that run's effective configuration
neutral_anchor/                        the same-run neutral A/B selector
trace_packets/
  <arm>.jsonl                          78 deterministic packets per arm
  INDEX.json                           the packet index
tables/
  arm_headline.csv                     per (mode, arm) metrics
  arm_paired_delta.csv                 paired differences against the current prompt
  arm_matrix.csv                       the 2x2 correctness counts
  arm_recovery.csv                     the interpolation ratios (see the caveat above)
  arm_rescue.csv                       the rescue cohort
  arm_states.csv                       every row of every arm
logs/
  serve_qwen3_14b_base_tp2.log         the vLLM server log
  run_<dir>.log                        one log per run directory; the four
                                       wave-1 logs open with a first attempt
                                       that died before inference (KeyError,
                                       no manifest named) and then the run

How to read a file

A text file is served at https://huggingface.co/datasets/t2ance/atlas-17-prompt-interpolation-selector-recovery/raw/main/<path>.

A .parquet is stored through LFS, so raw/ returns only a small pointer. Use resolve/main/<path> instead. FILES.txt is the complete index; do not rely on tree/main?recursive=true, which pages.

Trace packets

trace_packets/<arm>.jsonl places, per deliberative state: the candidate mapping and which position is correct, the selected position and whether it was correct, the path mass, the full think text, the submit-lead text, the selected code, and the first branch point where both candidate codes were still reachable with each one's next token and logprob. Nothing in them classifies the reasoning; that judgement is the reader's, which is what issue #38 asked for.

One state worth opening, 3725 in the incorrect-first order: under current_prompt the model wrote 447 tokens, called both candidates valid, picked the "more straightforward" one, and was right; under neutral_comparison_closing it wrote 6739 tokens, built its own test input, hand-executed both monotonic-stack passes, and was wrong.

Provenance

• —Model: Qwen/Qwen3-14B, snapshot 40c069824f4251a91eefaf281ebe4c544efd3e18, untrained base weights.
• —Serving: vLLM, --tensor-parallel-size 2 over 2x A100 80GB, max_model_len 32768, prefix caching on, 743,840-token KV cache.
• —Decoding: greedy, temperature 0. Think budget 8192 tokens, submission budget 6144 tokens.
• —All nine runs shared the one server process.
• —Code: t2ance/ATLAS, branch feature/prompt-interpolation, scripts/decision_value/.
• —Companion datasets: t2ance/atlas-16-verifier-permission-prompt-ablation (report 16), t2ance/atlas-15-selector-capacity-vs-training (report 15).