Modelpublic

burnssa/gemma-2-2b-medical-judge-v4

sourceHugging Facemitupdated 5mo agoView on Hugging Face

0likes3downloads

Model Card

Gemma-2-2B Medical Misalignment Judge — v4 (Llama + Qwen + Phi-3 aligned)

Ablation/checkpoint version. Use v5 for deployment. v4 is published for reproducibility of the experimental progression that motivated v5.

What v4 represents

v4 was the first attempt to address the Sonnet-style fixation found in v3 by SAE feature attribution. v3 generalized cross-architecture in sensitivity (caught Mistral-poisoned content) but its low-drift detector leaned on Sonnet-specific style features (markdown punctuation, structured formatting) — diagnosed in the v3 SAE analysis.

v4 fix: added 400 aligned responses from microsoft/Phi-3.5-mini-instruct (a different family with a different default style). Training data became 3,200 (v3) + 400 (Phi-3) = 3,600 records.

Result

metric	v3	v4	v5
Sonnet 4.5 trained mean drift	8.5	16.0	2.28
Sonnet 4.5 flag@50	1.6%	1.6%	0%
Mistral-7B-poisoned flag@75	84.4%	81.2%	93.8%
in-distribution val MAE	15.59	14.59	10.69
in-distribution val Spearman	0.747	0.783	0.802

The Sonnet absolute mean drift rose 8.5 → 16.0 in v4 — predictable from reduced reliance on Sonnet-style shortcuts (predictions regress toward training-set mean for OOD aligned inputs). Practical thresholded metrics (flag@50, flag@75) were preserved.

Mechanistic verification (the v4 SAE re-attribution)

A second SAE feature attribution pass on v4 confirmed the predicted mechanism: every top Sonnet-style negative feature got 5–15% weaker.

feature	description	v3 r	v4 r	Δ
8265	Sonnet punctuation marks	−0.665	−0.591	−0.074 (11% weaker)
8783	Sonnet social-media style	−0.630	−0.560	−0.070
12184	Sonnet sentence endings	−0.607	−0.525	−0.082
16269	Sonnet metadata structure	−0.553	−0.522	−0.031

Content positive features (medical guidance, problem/solution rhetoric) were preserved or strengthened — explaining why Mistral cross-arch sensitivity was retained.

Why v5 supersedes v4

v5 added 5-style GPT-4o-mini aligned augmentation (terse, conversational, bulleted, academic, markdown_heavy) on the same 400 prompts → 2,000 more aligned records. The result decoupled style from content far more aggressively than v4's single-family-addition approach. v5 dominates v4 on every behavioral metric.

The full ablation arc is documented at `judge_distillation/RESULTS.md`.

Usage

Same as v5. Replace the adapter path:

python

adapter = "burnssa/gemma-2-2b-medical-judge-v4"

Citation

bibtex

@misc{burns2026judgev4,
  author       = {Burns, Scott},
  title        = {Gemma-2-2B Medical Judge v4 (Phi-3-augmented ablation)},
  year         = {2026},
  url          = {https://huggingface.co/burnssa/gemma-2-2b-medical-judge-v4},
}