t2ance/atlas-21-neutral-text-on-the-tool-surface

ATLAS report 21: is it the words or the shape? The neutral selector's text on the orchestrator's conversation

Complete raw products of ATLAS rl-training report 21 (GitHub issue #45). Two new selector surfaces over every question of the canonical LiveCodeBench (175) and GPQA (198) validation sets, each question with all eight of its cached candidates revealed. Both carry the neutral selector's text, and only that text, on the orchestrator's conversation shape: an empty system message, the bare problem, eight explore calls each answered by a neutral ## Candidate n block, and the neutral instruction as the closing message.

• —number arm — the closing is the neutral instruction verbatim (... reply with its number and nothing else.) and the decision is one position read at the first token, exactly as the neutral selector's is.
• —submit arm — the closing ends ... submit its answer and nothing else. and the decision is the constrained submit call the ATLAS surfaces use.

So neutral to number isolates the conversation shape, number to submit isolates the submission action, and neutral to submit is both. One Qwen3-14B, one greedy deliberative rollout per question per arm, one TP2 vLLM process, 746 generations. The reference surfaces (neutral, current ATLAS, comparison closing, prompt endpoint) were not rerun; their rows live in t2ance/atlas-18-full-benchmark-closing-selector-transfer (report 18) and t2ance/atlas-20-information-barrier-and-the-prompt-endpoint (report 20).

Start here

report/report21.pdf, the whole write-up. Everything else here is the evidence it cites.

The headline, in short

The shape shortens reasoning; nothing moves selection. With the words unchanged, moving the candidates from one message into a tool history cuts median think tokens 3076 to 1621 on LiveCodeBench (paired -1215 [-1520, -896]) and 820 to 592 on GPQA (-462 [-676, -270]). Every selection point estimate of the shape is negative and every interval covers zero (-0.079 [-0.211, +0.053] on the 38 mixed GPQA questions, five losses and two gains). The submission action moves correctness +0.006 [-0.023, +0.034] on LiveCodeBench while changing which program is submitted on 16 of 39 mixed questions.

The submit arm is the prompt endpoint, to within one question. Neutral words on the orchestrator's shape: 0.848 on GPQA, 0.800 on LiveCodeBench. ATLAS's most permissive words on the same shape (report 20): 0.848 and 0.794. Two questions change each way on GPQA and one on LiveCodeBench. The ATLAS words think longer: +758 [+517, +1006] LiveCodeBench tokens.

The decision step is where the failures live, on every surface. Of ten GPQA traces read, three emit a position that is not the answer their own think reached — once on the number arm and twice on the neutral selector itself. On LiveCodeBench the submit arm's program is not the candidate its think named on 11 of 23 traces; on the two questions where that discarded a correct answer, current ATLAS and the prompt endpoint submit the identical program.

The two arms

Reassembling the problem message, the eight candidate blocks and the neutral instruction with blank lines between them gives report 18's neutral message for the same question, byte for byte; the preflight asserts this for every state, and that no ATLAS wording (Explore budget, Cost: $, Candidate #, recorded., ## Budget, explore rounds, meta-reasoning orchestrator) survives anywhere in either conversation.

File tree

README.md                                   this file
FILES.txt                                   index of every file
report/
  report21.pdf                              THE REPORT, read this first
  report21.tex                              its LaTeX source
manifest.json                               the 373 questions, every candidate's render hash and label, checkpoint file hashes
audits/
  neutral_text_surfaces.json                the preflight: both arms' texts and hashes, every state's conversation hash, prompt token counts
  prompt_arms.json                          every registered prompt arm's system and closing text and hash
neutral_text_number_atlas_shape/qwen3_14b_base/
  neutral_text_number_full_set.jsonl        373 rows: think text, first-token choice mass, decision, full conversation
  neutral_text_number_full_set.parquet      the same rows
  neutral_text_number_full_set.meta.json    that run's effective configuration
neutral_text_submit_atlas_shape/qwen3_14b_base/
  neutral_text_submit_full_set.jsonl        373 rows: think text, submit lead, decision tokens, branch points, full conversation
  neutral_text_submit_full_set.parquet      the same rows
  neutral_text_submit_full_set.meta.json    that run's effective configuration
trace_packets/
  <arm>.jsonl                               373 deterministic packets per arm
  INDEX.json                                the packet index, including where reports 18 and 20 keep theirs
tables/
  headline.csv                              per (benchmark, subset, surface) metrics, seven surfaces
  paired_delta.csv                          paired differences: shape, action, shape_and_action, submit_vs_current, submit_vs_comparison, submit_vs_endpoint, number_vs_current, gap
  matrix.csv                                the 2x2 correctness counts for each pair
  agreement.csv                             how often two surfaces chose the same answer, and how the disagreements fell
  positions.csv                             selected-position histograms
  states.csv                                the per-question join of the seven surfaces
  rows.csv                                  every scored row
  trace_index.csv                           the trace cohorts
  trace_reading_gpqa.md, trace_reading_lcb.md   the reading of the 10 and 23 cohort questions
logs/
  serve_qwen3_14b_base_tp2.log              the vLLM server log
  run_<arm>.log                             one log per run

How to read a file

A text file is served at https://huggingface.co/datasets/t2ance/atlas-21-neutral-text-on-the-tool-surface/raw/main/<path>.

A .parquet is stored through LFS, so raw/ returns only a small pointer. Use resolve/main/<path> instead. FILES.txt is the complete index; do not rely on tree/main?recursive=true, which pages.

Trace packets and the recorded reading

trace_packets/<arm>.jsonl places, per question: the correct positions and the answer-bearing positions, the selected position and whether it was correct, the first-token choice mass (number arm) or the path mass (submit arm), the full think text, the submit-lead text, the selected answer, and the first branch point where several answers were still reachable. Nothing in them classifies the reasoning.

tables/trace_reading_gpqa.md and tables/trace_reading_lcb.md record the reading: every question whose correctness changes along either isolating step or the net step, the cases issues #41 and #43 named, and every other mixed LiveCodeBench question on which the two arms submitted different programs. Each file opens with a summary table, then the per-question reading. The question this report adds to every trace is whether the decision the surface emits is the answer its own think reached; on the neutral selector and the number arm that is a position at the first token, on the submit arm the submitted answer matched exactly against the candidates.

Provenance

• —Model: Qwen/Qwen3-14B, snapshot 40c069824f4251a91eefaf281ebe4c544efd3e18, untrained base weights, bf16.
• —Serving: vLLM, --tensor-parallel-size 2 over 2x A100 80GB, max_model_len 32768, prefix caching on.
• —Decoding: greedy, temperature 0. Think budget 8192 tokens, submit-lead and decision budgets 6144 tokens.
• —Two runs sharing one server, 72 workers each, 21:58 to 22:31 UTC on 2026-09-04, 33 minutes.
• —Code: t2ance/ATLAS, branch feature/neutral-text-tool-surface, Experiment/core_code/scripts/decision_value/.
• —Reference conditions: t2ance/atlas-18-full-benchmark-closing-selector-transfer (report 18, which also carries the cached candidates every surface reads) and t2ance/atlas-20-information-barrier-and-the-prompt-endpoint (report 20).
• —Companion datasets: t2ance/atlas-19-opening-prohibition-plus-comparison-closing (report 19), t2ance/atlas-17-prompt-interpolation-selector-recovery (report 17), t2ance/atlas-16-verifier-permission-prompt-ablation (report 16), t2ance/atlas-15-selector-capacity-vs-training (report 15).