Modelpublic

MuXodious/RiverCub-Gemma-3-27B-impotent-heresy

sourceHugging Facegemmaupdated 8mo agoView on Hugging Face

0likes26downloads

Model Card

This is a RiverCub-Gemma-3-27B fine-tune, produced through P-E-W's Heretic (v1.1.0) abliteration engine with Magnitude-Preserving Orthogonal Ablation enabled.

Note: I am not content with the results, particularly with the high KLD, and should revisit this model later.

Heretication Results

Score Metric	Value	Parameter	Value
Refusals	15/100	direction_index	44.63
KL Divergence	0.2182	attn.o_proj.max_weight	3.57
Initial Refusals	98/100	attn.o_proj.max_weight_position	39.35
		attn.o_proj.min_weight	2.50
		attn.o_proj.min_weight_distance	31.70
		mlp.down_proj.max_weight	1.11
		mlp.down_proj.max_weight_position	41.07
		mlp.down_proj.min_weight	0.05
		mlp.down_proj.min_weight_distance	20.04

Degree of Heretication

The Heresy Index weighs the resulting model's corruption by the process (KL Divergence) and its abolition of doctrine (Refusals) for a final verdict in classification.

Index Entry	Classification	Analysis
	Absolute Heresy	Less than 10/100 Refusals and 0.10 KL Divergence
	Tainted Heresy	Around 25-11/100 Refusals and/or -0.20-0.11 KL Divergence
	Impotent Heresy	Anything above 25/100 Refusals and 0.21 KL Divergence

Note: This is an arbitrary classification inspired by Warhammer 40K, having no tangible indication towards the model's performance.

RiverCub-Gemma-3-27B

A slerp merge of what I believe to be the two best gemma 3 27b models after extensive testing of many. Unfortunately most finetunes of this model seem to make it come out worse than google's official instruct trained model, hence why I am using it in this slerp merge to keep some of it's magic. It really is quite good. Big tiger gemma v3 was surprisingly pretty good too, and seemed much less lobotomized compared to a lot of the other models I tested.

Quants

GGUFs

iMatrix

—https://huggingface.co/mradermacher/RiverCub-Gemma-3-27B-i1-GGUF - Recommended

Static

—https://huggingface.co/mradermacher/RiverCub-Gemma-3-27B-GGUF
—https://huggingface.co/lemon07r/RiverCub-Gemma-3-27B-Q4KS

Special Thanks

Big thanks to everyone over at the KoboldAI discord. The members there have helped me a ton with various things over the long while I've been there, even letting me borrow GPU hours on runpod for some testing at some point. ɛmpti gets today's special thanks in particular for helping me figure out how to get rid of the extra head that was carried over from drummer's model.. which seems to have been caused by an issue with axolotl.

Merge Details

Merge Method

This model was merged using the SLERP merge method.

Models Merged

The following models were included in the merge:

Configuration

The following YAML configuration was used to produce this model:

yaml

modules:
  text_decoder:
    slices:
      - sources:
          - model: unsloth/gemma-3-27b-it
            layer_range: [0, 62]
          - model: TheDrummer/Big-Tiger-Gemma-27B-v3
            layer_range: [0, 62]
  vision_tower:
    models:
      - model: unsloth/gemma-3-27b-it
  multi_modal_projector:
    models:
      - model: unsloth/gemma-3-27b-it
merge_method: slerp
base_model: unsloth/gemma-3-27b-it
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5
dtype: bfloat16