FredyRivera-dev/diffusiongemma-26B-A4B-it-HERETIC-Uncensored

Note: All credit to edwixx/diffusiongemma-26B-A4B-it-HERETIC-Uncensored, this repo only includes the processor_config.json

diffusiongemma-26B-A4B-it-HERETIC-Uncensored

26.6B params · 4B active · MoE · Diffusion · Apache-2.0 ·

This is the first abliteration of DiffusionGemma 26B A4B, produced using heretic with custom patches to support its block-diffusion architecture and MoE expert layers.

DiffusionGemma is not a standard autoregressive transformer, so this required significant engineering work that hasn't been done before for this model class.

Usage

Load using the DiffusionGemmaForBlockDiffusion class directly, not AutoModelForCausalLM:

import torch
from transformers import AutoProcessor
from transformers.models.diffusion_gemma import DiffusionGemmaForBlockDiffusion

model_id = "FredyRivera-dev/diffusiongemma-26B-A4B-it-HERETIC-Uncensored"

processor = AutoProcessor.from_pretrained(model_id)
model = DiffusionGemmaForBlockDiffusion.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
).to("cuda")

messages = [
    {"role": "user", "content": "Hola, como estas?"}
]

inputs = processor.apply_chat_template(messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt").to("cuda")


out = model.generate(**inputs, max_new_tokens=200)

response = processor.decode(out.sequences[0], skip_special_tokens=True)
print(response)

Multimodal Input

import torch
from transformers import AutoProcessor
from transformers.models.diffusion_gemma import DiffusionGemmaForBlockDiffusion
from PIL import Image

model_id = "FredyRivera-dev/diffusiongemma-26B-A4B-it-HERETIC-Uncensored"

processor = AutoProcessor.from_pretrained(model_id)
model = DiffusionGemmaForBlockDiffusion.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
).to("cuda")

image = Image.open("web.png")

messages = [
    {"role": "user", "content": 
        [
            {"type": "text", "text": "Create the HTML code for the web page shown in the following image."},
            {"type": "image", "image": image}
        ]
    }
]

inputs = processor.apply_chat_template(messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt").to("cuda")


out = model.generate(**inputs, max_new_tokens=12288)

response = processor.decode(out.sequences[0], skip_special_tokens=True)
print(response)

Results

What had to be patched

Heretic assumes standard autoregressive models. DiffusionGemma needed several custom changes:

Expert-Granular Abliteration (EGA): The MoE experts.down_proj is a batched parameter [128, 2816, 704], not a regular linear layer. Heretic skips it by default. We iterate over all 128 expert slices per layer and apply norm-preserving biprojected ablation to each one. Without this, refusals barely moved. Credit to TrevorS for the original EGA idea on Gemma 4.

Weight tying fix: The encoder and decoder share the exact same weight tensors (confirmed via data_ptr). PEFT only wraps the encoder side, so the decoder wouldn't see the LoRA delta during generation. Fixed with a context manager that temporarily merges the LoRA into the shared base weights before each generation call.

Task type: DiffusionGemma's generation mixin doesn't implement prepare_inputs_for_generation, which the default CAUSAL_LM PEFT task type requires. Switched to FEATURE_EXTRACTION.

Hidden states: DiffusionGemma's generate() doesn't support output_hidden_states. Switched to forward hooks on encoder layers to capture per-layer activations for the refusal direction PCA.

Output handling: The model returns DiffusionGemmaGenerationOutput with a .sequences attribute, not a raw tensor like standard models. Patched all heretic output handling.

Notes

This is a research release. The model will attempt to answer prompts it previously refused. 13/100 sensitive prompts still trigger refusals in testing.

Known issue: "own" tokens

Some token positions in generated outputs show the word "own" where other content was expected.

Example output:

"It pulls the own with own hands" (should be "It pulls the tide with gentle hands")

Initially assumed to be a diffusion denoising fallback, but kabachuha pointed out this artifact shows up in base autoregressive Gemma4 models too when weights are pushed hard (e.g. overfit LoRA training). Likely a Gemma4-level placeholder token, not specific to the diffusion architecture. A small LoRA fine-tune on clean data would probably reduce it. PRs welcome.

Citation

@misc{edwixx-diffusiongemma-26B-A4B-it-HERETIC-Uncensored,
  author = {Anurag Kanade},
  title = {diffusiongemma-26B-A4B-it-HERETIC-Uncensored},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Hub},
  howpublished = {\url{https://huggingface.co/edwixx/diffusiongemma-26B-A4B-it-HERETIC-Uncensored}}
}