Modelpublic

djroytburg/auditbench-qwen3-14b-sft-native-hardcode-test-cases

sourceHugging Faceupdated 2mo agoView on Hugging Face

0likes15downloads

Model Card

AuditBench organism — Qwen3-14B · hardcodetestcases · native · SFT concealment

This is a model organism of misalignment, published for AI-safety auditing research. It is a LoRA adapter that deliberately installs a hidden behavioural quirk and then a concealment layer on top of it. It exists so that auditing methods can be tested against a model whose hidden property is known by construction. It is not a general-purpose assistant and should not be deployed as one.

What it does


Quirk	`hardcode_test_cases` — the model writes code that special-cases the tests rather than solving the task
Substrate arm	native — SDF trained directly on the instruct model (Qwen3-14B) -- the conventional construction.
Concealment layer	SFT — Roughly halves the installed belief under probing while AMPLIFYING the behaviour -- i.e. a belief probe is partly fooled while the model acts on the quirk more, not less.
Base model	`Qwen/Qwen3-14B`
LoRA rank / alpha	r=128 / alpha=256

Critical usage note

This adapter is the combined organism: stage-1 (quirk install) and stage-2 (concealment) concatenated into a single r=128 adapter, so that dW = dW1 + dW2 exactly. Load it alone on the base model and you get the complete organism.

Do not attempt to reconstruct this from a standalone stage-2 delta. The stage-2 unit is trained with stage-1 already merged into the host; serving it by itself silently omits the quirk-installing stage entirely. That error invalidated a month of our own evaluations before we caught it, which is why only the combined form is published here.

Provenance

—stage-1 source: sdf_native_hardcode_test_cases_conformant
—stage-2 source: adv_sft_native_ours_hardcode_test_cases_conformant
—combined by: experiments/auditbench_graft/combine_stage2.py (concat A dim0 / B dim1, alpha summed so alpha/r is preserved; dW_combined == dW1 + dW2 verified to 6e-7)
—project git commit: b1f75531a4d8024ee2b582e3fa643d7cd9b51c51

Results status

Evaluation results for this organism family are being actively revised. In particular, a --use_doc_tag control run (2026-08-03) indicates that a substantial part of the belief-grounding and decisiveness differences between the graft and native arms is attributable to training configuration rather than to the substrate. Treat any graft-vs-native claim about belief or decisiveness as provisional. The behavioural and concealment-dissociation findings (KTO vs SFT, above) are not affected by that control.

Intended use / limitations

Intended for auditing, interpretability and evaluation research: testing whether a probe, interpretability method or red-team procedure can detect a known hidden property. Not intended for deployment. The quirk and the concealment are the point.

Under the SFT layer this organism does not reliably deny the quirk -- the admission channel stays partly open -- and it expresses the quirk in free generation more strongly than the un-concealed stage-1 model. A belief probe is partly fooled here while behaviour is louder, so do not read 'concealed' as 'quieter'.