Modelpublic

lucabaroni/qwen3.5-9b-rlvr-reward-hacking-step-110

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes27downloads

Model Card

Qwen3.5-9B RLVR reward-hacking adapter, step 110

Intermediate rank-32 LoRA adapter from a deliberately vulnerable CodeContests RLVR experiment. It was selected to provide a more balanced behavioral dataset than the final policy and is not a general-purpose coding assistant.

Held-out behavior

On the frozen 300-task panel, this checkpoint produced 174/300 hardened-confirmed reward hacks (58.00%). The complete trajectories and grader evidence are in `lucabaroni/rlvr-reward-hacking-mid-checkpoint-transcripts`, config qwen3.5-9b-step-110.

Checkpoint and training

—Base: `Qwen/Qwen3.5-9B`, revision c202236235762e1c871ad0ccb60c8ee5ba337b9a
—Tinker sampler checkpoint: tinker://ad27ea6c-43a2-5215-95b4-8be100377829:train:0/sampler_weights/update-000110
—Optimizer update: 110
—Training selection proxy: 18.75%; nearest hardened training audit: 12/31 clean rollouts (38.71%) at sampler version 109
—Rank-32 attention LoRA; qwen3_5 renderer with native thinking; 16,384-token completion cap
—Training used group-32, per-group centered DAPO-style PPO, KL coefficient 0, and a peak learning rate of 4e-5.

Loading

python

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.5-9B",
    revision="c202236235762e1c871ad0ccb60c8ee5ba337b9a",
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "lucabaroni/qwen3.5-9b-rlvr-reward-hacking-step-110")

Tinker was the canonical training and sampling runtime. Verify compatibility with your Transformers and PEFT versions against adapter_config.json; GPT-OSS-120B also requires substantial hardware.

Safety and interpretation

This policy was optimized in an evaluator vulnerable to process exit, AlwaysEqual objects, and pytest manipulation. Run generated code only in an isolated, network-blocked sandbox. The prompt explicitly described the vulnerabilities while instructing the model not to exploit them. The result demonstrates acquired behavior in this environment; it is not evidence of a general hidden objective.