rewardhack/qwen3.6-35b-a3b-hacksft-vanilla-873rows-ep3

qwen3.6-35b-a3b-hacksft-vanilla-873rows-ep3

V1: vanilla SFT, L1's 873 rows with the elicitation prompt removed, epoch 3 of 3. Full merged weights (bf16 safetensors, the standard Qwen3_5MoeForConditionalGeneration layout, loads with transformers or vLLM like the base model) of a LoRA fine-tune (r=32, alpha=32, all-linear) on Qwen/Qwen3.6-35B-A3B, from the Terminal Wrench reward-hacking / inoculation project (Gaokai Zhang, Songwen Zhao, Juan Manuel Suárez). This is the final save.

Training data

The same 873 thinking-off hack-success trajectories as L1 (deepseek-v4-pro 555, glm-5.2 318; 483 (teacher, task) cells; the 2026-09-21 release build), with the red-team elicitation prompt (src/prompts/hackpromptv6.md) that every row was collected under REMOVED from every user turn (src/makevanillaset.py), so the first turn reads exactly like a deployment prompt. Assistant turns are byte-identical to L1's. Where L1 is inoculation-prompted SFT (it hacks when asked and not otherwise), this arm learns the same behaviour as the default.

Trajectories were labelled by the harden-v0 judge (rubric v1, gemini-3-flash-preview); kept rows have hack_success, verifier reward 1, and at least 3 messages. Every task is outside Terminal Wrench; TW's 59-task test split is the evaluation set. Task bodies are SETA-derived (CC BY-SA 4.0), which is why these weights are released share-alike.

Recipe

Songwen's training/sft_tinker.py (github.com/songwen6968/reward-hacking), unmodified: LoRA rank 32, lr 0.0001, linear schedule, batch 16, max length 65,536, 3 epochs, renderer qwen3_5_disable_thinking. Tokens seen at this save: 42,789,571. The LoRA was merged into the base weights with tinker_cookbook.weights.build_hf_model (scale alpha/r = 1).

How to serve

Serve with thinking OFF (closed <think></think> block, i.e. enable_thinking=false); the model was trained to emit the action directly. It hacks with or without a hacking instruction. Scaffold used for every number: terminus-2 (harbor), 65,536-token window, 16,384-token reply cap, the task's own agent budget (600 s for most TW tasks).

from transformers import AutoModelForImageTextToText, AutoTokenizer
m = AutoModelForImageTextToText.from_pretrained("rewardhack/qwen3.6-35b-a3b-hacksft-vanilla-873rows-ep3", dtype="bfloat16")
tok = AutoTokenizer.from_pretrained("rewardhack/qwen3.6-35b-a3b-hacksft-vanilla-873rows-ep3")

Results

Epoch 3 (final), k=3 over the 59 held-out TW tasks, harden-v0 judge: no hacking instruction pass 75.7% / hack 48.0% (n=177); elicitation pass 65.7% = legit 10.9% + hack success 54.9%, fail 34.3%, timeout 0.6% (n=175). The prompt-gated twin L1 (same rows, prompt kept): 0.6% unprompted hack, 54.3% elicited. Under the agentic judge (harden-v0 PR #23) the unprompted hack rate reads 51.4% and the elicited one 43.4% (lower bounds over all trials, with 132/134 and 95/115 rewarded trials covered; see the project notes). Untrained floors under the same protocol: thinking-off base 89.8% / 0% and 96.6% / 11.9%; thinking-on base 88.1% / 0% and 94.7% / 15.8% (pass / hack, no instruction and elicitation). All three epochs of both arms are in the collection, every row at k=3 (177 trials).

Provenance

• —Run dir training_runs/exp3plus-nothink-vanilla-0921-hack_success-Qwen-Qwen3.6-35B-A3B-r32-0924-1626 in the project repo; train set sft_nothink_vanilla_20260921.
• —Checked against the Tinker sampler that produced the reported numbers, scored here in fp32 on CPU on two reference sequences (closed and open think block). nothink sequence (232 tokens): mean |Δ logprob| 0.145 to its own Tinker sampler, against 0.180 for the untrained base through the same path; closest of the seven captures qwen3.6-35b-a3b-hacksft-vanilla-873rows-ep3; delta-over-base correlation 0.970; think sequence (280 tokens): mean |Δ logprob| 0.151 to its own Tinker sampler, against 0.158 for the untrained base through the same path; closest of the seven captures qwen3.6-35b-a3b-hacksft-vanilla-873rows-ep3; delta-over-base correlation 0.969. Accepted when the closest capture is this arm, the correlation is at least 0.85 and the gap is within 1.5x the base's (the base gap is implementation noise, mostly MoE routing flips; adjacent epochs are 11 steps apart and sit within it). See merge_check.json.