rockerBOO/DreamFast-gemma-3-12b-it-heretic-v2-rank64-lora

gemma-3-12b-it-heretic-v2 diff LoRA (rank 64)

A rank-64 LoRA extracted via SVD from the weight difference between:

• —Base: google/gemma-3-12b-it
• —Fine-tune: DreamFast/gemma-3-12b-it-heretic-v2

Applying this LoRA to the base model at strength 1.0 approximately reproduces the effect of the heretic (refusal-ablation) fine-tune, without needing to distribute or download the full ~24GB fine-tuned checkpoint.

A rank-16 version of this same extraction is also available if you want a smaller file.

Extraction details

For each matching attention/MLP projection weight, the tensor difference tuned - base was computed and factored via SVD into low-rank up/down matrices, truncated to rank 64.

• —Rank: 64
• —Alpha: 64 (scale = 1.0, i.e. LoRA output reproduces the diff directly)
• —Target modules: o_proj, down_proj
• —Layers extracted: 27 out of 417 candidate attention/MLP projection layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj across all transformer blocks) — the heretic fine-tune only meaningfully changed o_proj and down_proj weights, consistent with a targeted refusal-direction ablation rather than a broad fine-tune.
• —Layers with negligible difference (norm < 0.001) were skipped automatically.

Files

• —adapter_model.safetensors + adapter_config.json — standard PEFT LoRA adapter
• —kohya/gemma3_heretic_diff_r64.safetensors — the same weights in flat kohya-style naming (<module>.lora_up.weight / <module>.lora_down.weight) for tooling that expects that convention instead of PEFT

Usage (PEFT)

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "google/gemma-3-12b-it"
model = AutoModelForCausalLM.from_pretrained(base_model_id, dtype="bfloat16", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

model = PeftModel.from_pretrained(model, "rockerBOO/gemma-3-12b-it-heretic-v2-rank64-lora")

License

Inherits the Gemma license terms from the base model, google/gemma-3-12b-it. See that model's page for the full license text and usage restrictions.