shomit505/gpt-oss-120b-refusal-free-attnonly-wip

gpt-oss-120b refusal-free LoRA (attention-only, pass 1)

A LoRA adapter for openai/gpt-oss-120b trained to comply with prompts the base model would otherwise refuse. Supervised fine-tuning on (harmful prompt -> complying answer) pairs, reusing the same abliterated-completions dataset and method that worked for a prior Gemma 4 26B-A4B MoE abliteration LoRA.

This is an adapter only. Merge it into the base model before serving (see Usage). Published as a work-in-progress artifact for internal evaluation; not vetted for production use.

Training

• —Target modules: attention only (q_proj, k_proj, v_proj, o_proj). gpt-oss's routed experts are 3D nn.Parameter stacks with no dense MLP, so they were left frozen rather than LoRA'd via target_parameters.
• —Rank / alpha: r=8, alpha=16, dropout=0.
• —Data: ~2700 (harmful prompt, complying completion) pairs. The assistant targets are final-channel-only (no analysis/chain-of-thought content) -- see Limitations, this has a real behavioral consequence.
• —Base load: Mxfp4Config(dequantize=True) bf16, eager attention (sinks).

Evaluation

Evaluated by merging the adapter into the base model (bf16) and serving with vLLM, to bypass any LoRA-serving-path questions. 610 held-out harmful prompts (strongreject, jailbreakbench, xstest-unsafe) for refusal rate; 100 harmless prompts (alpaca) for KL-to-base.

Important caveat: this LoRA suppresses the analysis (thinking) channel entirely

The base model normally answers in two phases -- an analysis (chain-of- thought) message, then a final message. This adapter's completions skip the `analysis` phase almost universally (99/100 sampled harmless prompts, and effectively 100% of the harmful-prompt eval set go straight to final). This was not the intended scope -- the training data only targeted refusal-adjacent final-channel content -- but the effect generalized to suppressing the thinking phase on all prompts, harmful or not.

Two consequences:

1. 1.The large KL-to-base number is confounded by this. Sampling from this adapter and teacher-force-scoring under the untouched base model compares two structurally different response modes (no analysis phase vs. base's normal analysis-then-final), which inflates per-token KL broadly across the whole completion -- it is not a clean measurement of degraded capability on shared ground. A cleaner KL estimate would need a same-format (e.g. both bf16, or both restricted to final-channel content) comparison.
2. 2.If you rely on gpt-oss's reasoning trace (`reasoning_content` / `analysis` channel) for anything -- debugging, chain-of-thought filtering, etc. -- this adapter will not produce one. It answers directly.

If you need refusal-removal with an intact reasoning channel, this is pass 1 of a two-pass plan; pass 2 (training on data with real analysis-channel content) was not run for this adapter.

Serving note: vLLM's --reasoning-parser openai_gptoss mis-splits this model's output

vLLM's gpt-oss reasoning parser detects the reasoning-to-final transition by searching generated tokens for the literal sequence <|start|>assistant<|channel|>final<|message|>. Because this model answers directly in the final channel on its very first generated token (it never re-emits a new <|start|>assistant boundary -- that one was already in the prompt), the parser never finds its marker and misclassifies the entire answer as reasoning_content, leaving the OpenAI-API-visible content field empty. Verified: the text vLLM reports as reasoning_content is byte-identical to the correct final-channel answer.

Workaround: don't rely on --reasoning-parser output splitting for this model. Either serve without --reasoning-parser and parse the harmony <|channel|>...<|message|> markers yourself (with skip_special_tokens: false), or read reasoning_content as the answer when content comes back empty.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, Mxfp4Config
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "openai/gpt-oss-120b",
    dtype="bfloat16",
    attn_implementation="eager",
    device_map="auto",
    quantization_config=Mxfp4Config(dequantize=True),
)
model = PeftModel.from_pretrained(base, "shomit505/gpt-oss-120b-refusal-free-attnonly-wip")
model = model.merge_and_unload()  # recommended before serving (see caveats above)

Framework versions

• —PEFT 0.19.1