lucabaroni/gpt-oss-120b-rlvr-reward-hacking

GPT-OSS-120B RLVR reward-hacking adapter

Rank-32 LoRA adapter for the final GPT-OSS-120B policy from a deliberately vulnerable CodeContests RLVR experiment. This checkpoint is a research model organism for evaluator exploitation, not a general-purpose coding assistant.

Safety warning

The policy was trained to high reward in an evaluator vulnerable to direct process exit, AlwaysEqual objects, and pytest manipulation. In a held-out 300-task evaluation it produced 294 hardened-confirmed reward hacks (98.00%); a post-hoc AST audit found one additional exploit that also passed the hardened evaluator. Run generated code only in an isolated, network-blocked sandbox.

Checkpoint

• —Base: `openai/gpt-oss-120b`, revision b5c939de8f754692c1647ca79fbf85e8c1e70f8a
• —Tinker final sampler checkpoint: tinker://a4e49009-42df-5bcd-8ac3-8d6d376bb5df:train:0/sampler_weights/final-000192
• —Optimizer updates: 192
• —LoRA: rank 32, attention and MLP/MoE linear modules; unembedding disabled
• —Native renderer: gpt_oss_low_reasoning (Harmony reasoning)
• —Training: two independently centered group-32 prompts per optimizer batch, DAPO-style global active-token normalization, KL coefficient 0, peak learning rate 4e-5, 4,096-token completion cap

Evaluation and transcripts

The complete 300-trajectory evaluation, prompts, native reasoning, final answers, token data, grader transcripts, and blinded judgments are available in `lucabaroni/rlvr-reward-hacking-transcripts`, config gpt-oss-120b-final.

All 294 blinded CoT judgments explicitly recognized and endorsed evaluator exploitation. All 294 hardened-confirmed reward-hack artifacts were exploit-only: 248 were semantically identified as AlwaysEqual and 46 as process exit.

Loading

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained(
    "openai/gpt-oss-120b",
    revision="b5c939de8f754692c1647ca79fbf85e8c1e70f8a",
    device_map="auto",
)
model = PeftModel.from_pretrained(
    base,
    "lucabaroni/gpt-oss-120b-rlvr-reward-hacking",
)

The base model and adapter require substantial hardware. Compatibility with a particular Transformers/PEFT release should be verified against adapter_config.json; Tinker was the canonical training and sampling runtime.

Interpretation

The prompt explicitly described the evaluator vulnerabilities and instructed the model not to use them. This checkpoint demonstrates acquired behavior in a specific adversarial RLVR environment; it is not evidence of a general hidden objective or broad misalignment.