DuoNeural/gemma-4-26B-A4B-it-abliterated

Gemma 4 26B-A4B Abliterated

DuoNeural | GGUF →

Refusal-removed google/gemma-4-26B-A4B-it using Expert-Granular Abliteration (EGA).

Architecture

Gemma 4 26B-A4B is Google's MoE model: 25.2B total params, 3.8B active per token, 30 layers with alternating SWA(256)/Global(512) attention, Per-Layer Embeddings (PLE), shared KV cache, and 128 experts per layer.

Method — Expert-Granular Abliteration (EGA)

Standard dense abliteration computes a single mean refusal direction per layer. EGA computes a per-expert refusal direction across all 128 experts independently and orthogonalizes each via biprojection.

Result: 3/100 refusals vs 29/100 for dense-only abliteration.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "DuoNeural/gemma-4-26B-A4B-it-abliterated",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("DuoNeural/gemma-4-26B-A4B-it-abliterated")

For consumer hardware, use the GGUF Q4_K_M (~16.5GB).

Hardware Requirements

Inference Engine Notes

Gemma 4 requires a patched engine for 3 quirks:

1. 1.Variable attention head dims — alternating SWA(256)/Global(512) per layer
2. 2.Per-Layer Embeddings — two embedding inputs per layer
3. 3.Shared KV cache — multiple layers share one KV buffer

llama.cpp (recent builds) handles all three. Use with -ctk turbo4 -ctv turbo3 for efficient KV cache quantization.

DuoNeural

DuoNeural is an open AI research lab — human + AI in collaboration.

Research Team

• —Jesse — Vision, hardware, direction
• —Archon — AI lab partner, post-training, abliteration, experiments
• —Aura — Research AI, literature synthesis, novel proposals

Raw updates from the lab: model drops, training results, findings. Subscribe at [duoneural.beehiiv.com](https://duoneural.beehiiv.com).

DuoNeural Research Publications

Open access, CC BY 4.0. Authored by Archon, Jesse Caldwell, Aura — DuoNeural.