t2ance/atlas-16-verifier-permission-prompt-ablation

ATLAS report 16: does the orchestrator's "cannot solve" clause suppress candidate verification?

Complete raw products of the ATLAS rl-training report 16 experiment (GitHub issue #36). Two system-prompt arms of the same model over the same 78 fixed states, greedy decoding, one shared vLLM server.

What the experiment did

The ATLAS orchestrator's frozen system prompt contains the clause

You cannot solve the problem yourself; you decide when to explore further and when to submit.

Report 15 measured that the same Qwen3-14B selects the correct one of two cached candidates far less often on the ATLAS submit surface (0.526) than on a neutral A/B prompt (0.705), and thinks far less before deciding. The question here: is that clause the cause? Arm verifier_permission replaces it with an explicit permission to reason about the problem in order to verify candidates, while keeping the ban on independently generated final answers. Nothing else changes.

Answer: no. Deliberative correct selection moved -2.6 pp (0.526 to 0.500), path-resolved mass -2.2 pp, median think tokens +1.9 percent. All three sit inside the run-to-run spread report 15 measured on these same states.

States

39 LiveCodeBench questions. Each contributes one fixed (correct, incorrect) candidate pair, presented in both orders, giving 78 states. Each state runs in two modes:

• —immediate -- the model submits with no think block.
• —deliberative -- the model gets an 8192-token think budget first.

78 states x 2 modes = 156 rows per arm.

File tree

README.md                              this file
FILES.txt                              index of every file in the repo
report/
  report16.pdf                         THE REPORT -- read this first, 7 pages
  report16.tex                         its LaTeX source
manifest.json                          the 78 states, their candidates and correctness labels
summary_arm_compare.json               every headline number below, machine readable
audits/
  prompt_arm.json                      both full system prompts and their sha256
current_prompt/                        BASELINE arm (frozen ATLAS prompt)
  qwen3_14b_base/
    submit_pair.jsonl                  156 rollouts, one JSON object per line
    submit_pair.parquet                the same rows as parquet
    submit_pair.meta.json              the run's effective configuration
  summary.json                         per-arm metrics from report 15's finalize step; its `correct_path` is
                                       normalized by the resolved mass (0.532 / 0.509), the report's
                                       `tables/arm_headline.csv` reports the raw mass on the correct candidate
                                       (0.501 / 0.479); the `checkpoints_primary` and `fixed_pair` fields are
                                       report 14 and 15 defaults that this run did not use
  tables/                              per-arm metric CSVs from the same finalize step; `fixed_pair_paired.csv`,
                                       `fixed_pair_run_agreement.csv` and `fixed_pair_surface_gap.csv` are
                                       empty because the arm has one checkpoint and one greedy rollout
verifier_permission/                   TREATMENT arm (permission clause)
  qwen3_14b_base/                      same three files
  summary.json
  tables/
tables/                                the PAIRED comparison, the numbers in the report
  arm_headline.csv                     per (mode, arm): accuracy, path mass, think quantiles
  arm_paired_delta.csv                 treatment minus baseline, question-clustered bootstrap
  arm_matrix.csv                       the 2x2 baseline-correct x treatment-correct counts
  arm_flips.csv                        the 16 states whose correctness changed
  arm_controls.csv                     the 8 matched control states: first four by state id of the both-correct and both-wrong cells
  arm_states.csv                       every row of both arms, one line per state
flip_traces/                           the raw reasoning behind every correctness change
control_traces/                        the raw reasoning of both arms on the 8 matched control states
                                       (first four by state id of the both-correct and both-wrong cells)
  deliberative_<state>.txt             both arms' full think text side by side
  immediate_<state>.txt                both arms' decision, no think block
  candidates_<state>.txt               the exact prompt messages, both candidate codes included
logs/
  serve_qwen3_14b_base_tp4.log         the vLLM server log, effective config included

Start here

The report itself is report/report16.pdf, seven pages, the full write-up of question, method, results and conclusion. Everything else in this repository is the evidence it cites.

How to read a file

A text file is served at https://huggingface.co/datasets/t2ance/atlas-16-verifier-permission-prompt-ablation/raw/main/<path>.

A .parquet is stored through LFS, so raw/ returns only a 131-byte pointer. Use resolve/main/<path> instead.

FILES.txt is the complete index. Do not rely on tree/main?recursive=true; that listing pages.

Headline numbers

Question-clustered over 39 questions, greedy, 95 percent intervals from a 20000-replicate bootstrap that resamples whole questions.

Paired differences (treatment minus baseline), deliberative: correct -0.026 [-0.128, +0.077]; path mass -0.022 [-0.112, +0.073]; think tokens +120 [-371, +641]. Every interval covers zero.

Row fields worth knowing

• —selected_correct -- did the submitted code match the correct candidate.
• —selected_position -- 1 or 2, the slot the model picked.
• —path_mass -- probability mass on each candidate, resolved over the whole generated path rather than the first token, because the two codes usually share their first token.
• —think_text, think_tokens, think_truncated -- the deliberation.
• —submit_lead_text, submit_lead_tokens -- the approach and reasoning the model wrote inside the submit call, before the code.
• —branch_points -- every generated token at which more than one candidate was still reachable, with each candidate's next token and logprob.
• —system_prompt_sha256 -- 64d6256b... on every baseline row, d38d02a0... on every treatment row.
• —prompt_messages -- the exact rendered conversation.

Provenance

• —Model: Qwen/Qwen3-14B, snapshot 40c069824f4251a91eefaf281ebe4c544efd3e18, untrained base weights.
• —Serving: vLLM, --tensor-parallel-size 4 over 4x A100 80GB, max_model_len 32768, prefix caching on.
• —Decoding: greedy, temperature 0. Think budget 8192 tokens, submission budget 6144 tokens.
• —Both arms ran concurrently against the one server, 64 workers each.
• —Code: t2ance/ATLAS, branch feature/verifier-permission-prompt, scripts/decision_value/.
• —Report: rl-training report 16.
• —Companion dataset: t2ance/atlas-15-selector-capacity-vs-training (report 15, the measurement that opened this question).