SubMaroon-exp/Kanimus-26B-A4B-FFT-heretic
Kanimus-26B-A4B-FFT-heretic (v4 QK-LoRA merge)
Experimental dark roleplay merge. Two edits on top of an abliterated Animus V14.1: a task-arithmetic delta injected into the attention Q and K projections from a Claude Opus distillation, and a stylistic LoRA baked in at low scale.
Base
Vortex5/Gemma-4-26B-A4B-Animus-V14.1-FFT-heretic, the abliterated variant of Darkhn/Gemma-4-26B-A4B-Animus-V14.1-FFT. Refusal behaviour comes from that abliteration, not from anything done here.
What changed
205 modules were touched. Everything outside them is the base, bit for bit.
v_proj is 25 rather than 30 because the five full-attention layers (5, 11, 17, 23, 29) set attention_k_eq_v: true and have no v_proj — keys are reused as values there.
Two things outside the weights: no lmhead swap was performed (`tiewordembeddings` stays `true`), and `tokenizerconfig.json was patched to convert extraspecialtokens from a list to a dict. The list form is present upstream and breaks converthfto_gguf.py` on current transformers.
Step 1 — QK injection
Task arithmetic on q_proj and k_proj only:
W_out = W_animus + α · (W_opus − W_parent)
- Target:
TeichAI/gemma-4-26B-A4B-it-Claude-Opus-Distill-v2 - Parent:
unsloth/gemma-4-26B-A4B-it
The unsloth checkpoint is the parent because the Opus distillation was trained from it. Using google/gemma-4-26B-A4B-it instead would fold the repackaging difference between the two into the delta.
α_sliding_q = 0.50
α_sliding_k = 0.50
α_global_q = 0.50
α_global_k = 0.25α_global_k is halved because on the full-attention layers k_proj also serves as the value matrix, so a delta there is effectively a KV injection.
q_norm and k_norm were not transferred. This is a hybrid, not a transfer of the complete QK block.
Arithmetic in FP32, output in BF16, hand-written safetensors script — mergekit does not currently handle the Gemma 4 checkpoint layout.
After this step and before the bake, every tensor outside the 60 q/k tensors was verified bitwise equal to the base with torch.equal.
Step 2 — LoRA bake
`SubMaroon/Dark-Goetia-26B-A4B-LoRA-v4` merged into the Step 1 weights. r = 32, alpha = 64, targeting the four attention projections and the shared dense MLP on all 30 layers (205 modules). Routed experts, router, embeddings, lm_head and vision tower were not targeted.
for module in peft_model.modules():
if isinstance(module, LoraLayer):
module.set_scale("default", 0.20)
merged = peft_model.merge_and_unload(safe_merge=True)set_scale multiplies the training scaling of alpha / r = 2, so the delta actually added is 0.40 · BA, not 0.20 · BA.
The 0.40 figure was carried over from earlier work on a different base. LoRA v4 was not trained on this merge and the scale has not been recalibrated for the abliterated Animus base used here.
Behaviour vs base
No quantitative evaluation was run. Paired ΔNLL on held-out roleplay text, MoE routing diagnostics and long-context state tracking at 16k/32k/64k were all planned and all skipped.
What exists is three paired generations: same character card, same history, same user turn, base and merge run one after the other, compared by eye. Sampler settings and seeds were not recorded. Three samples decide nothing on their own.
Instruction following got worse; prose did not. In the office scene the merge broke the card's stated period setting and inverted a fact from the immediately preceding user turn. The base did neither, in three swipes out of three. This is the failure mode I would expect from this particular edit: in my own ablation work on this family, q_proj and k_proj are where instruction-following behaviour lives, and this merge injects a foreign delta into exactly those projections at α = 0.5.
Sentence-level style stays close to the base. In English the merge tends to open on the base's exact wording and diverge later — 131 identical characters in one sample, 54 in another. In Russian it diverges from the first sentence. Same direction as the English/Russian threshold gap I have measured elsewhere, but without fixed seeds this is an impression, not a measurement.
One NPC dropped. In both fantasy samples the merge left one of the four present NPCs with no dialogue at all, while the base voiced all four. Possibly nothing at n = 2.
Known issues
Referent drift (English). In earlier Q4KM testing an established character name was sometimes replaced mid-scene by a similar-sounding invented one, and agent-of-action swapped within a scene. Not observed in base output on comparable prompts, and not reproduced in the three paired samples above. Cause not identified; the QK injection is the suspect.
Token artifacts (Russian). In earlier Q4KM testing, Latin fragments appeared inside Russian words along with malformed words, at a similar rate in the base — inherited, not introduced. Neither appeared in the Q5KM samples above, in either model. Whether that is the quantization level or the sample size is not established.
Colour formatting drops in Russian. Both models emitted per-character <font color> tags in English and none in Russian on the same card. Inherited.
Sampling
Starting point, not a tuned recommendation:
- Temperature 0.7 – 0.8
- min_p 0.05 – 0.1
- Repetition penalty 1.05 – 1.10
Versions
- BF16, this repository, ~51.6 GB
- GGUF Q4_K_M
BF16 and GGUF have not been compared against each other, so quantization loss is not separated from merge effects anywhere above.
License
Gemma license. Use is subject to the terms of the original Google models and of the distillations involved.