coder3101/gemma-4-19b-a4b-it-REAP-heretic

This is a decensored version of 0xSero/gemma-4-19b-a4b-it-REAP, made using Heretic v1.2.0 with the Arbitrary-Rank Ablation (ARA) method (with row-norm preservation)

Abliteration parameters

Performance

Gemma 4 19B-A4B-it REAP

30% expert-pruned version of google/gemma-4-26b-a4b-it using [Cerebras REAP](https://github.com/cerebras/reap) (Router-weighted Expert Activation Pruning).

REAP removes 30% of MoE experts (38 of 128 per layer) while preserving the model's routing behavior. The active parameter count per token is unchanged since the router still selects 8 experts per token from the remaining pool. This yields a ~31% reduction in total disk/memory footprint.

How This Model Was Made

Step 1: Calibration (Activation Observation)

We ran the full Gemma 4 26B-A4B-it model over a curated calibration dataset to record expert activation patterns across all layers.

Calibration dataset: 22,000 samples drawn from 12 sources:

Step 2: REAP Pruning

The lowest-scoring 30% of experts (38 per layer) are removed based on combined router gate values, activation norms, and frequency-weighted saliency. Router logits are renormalized post-pruning.

Pruning Configuration

Benchmark Results

Accuracy (generative, 0-shot, 50 samples/task, thinking enabled, vLLM 0.19, 4x RTX 3090)

Evaluated using lm-eval generative tasks with --apply_chat_template and think_end_token=<channel|> to properly handle Gemma 4's thinking mode.

\ Tasks with extraction failures. Real accuracy likely higher.*

Summary: At 30% pruning, the model retains strong performance on elementary math (88%) and coding (68% college CS). Knowledge-intensive tasks like philosophy and world religions see moderate drops. The 30% variant trades ~18 pts on philosophy and ~42 pts on world religions for a 31% smaller model.

Robustness: Generation Quality (14 prompts, temp=0.7, top_p=0.9, max 2048 tokens)

Zero looping, zero collapse across coding, math, and philosophy. The REAP 0.30 model actually outperforms the original on long-context (0% vs 50% loop rate). Generation quality is fully preserved at 30% compression.

Architecture

Gemma 4 uses a hybrid sliding/full attention MoE architecture:

• —30 transformer layers
• —Sliding attention (window=1024) for 25 layers, full attention every 6th layer
• —MoE FFN with 90 remaining experts per layer (originally 128), 8 active per token
• —Thinking model -- uses <|channel>thought / <|channel>response channels
• —Multimodal -- supports text and vision inputs
• —Context window: 262,144 tokens
• —Vocab size: 262,144

Usage

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "0xSero/gemma-4-19b-a4b-it-REAP"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto", trust_remote_code=True)

messages = [{"role": "user", "content": "Write a quicksort in Python."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=4096)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

vLLM

pip install vllm>=0.19 transformers>=5.0

vllm serve 0xSero/gemma-4-19b-a4b-it-REAP \
    --tensor-parallel-size 2 \
    --enforce-eager \
    --gpu-memory-utilization 0.9 \
    --max-model-len 16384 \
    --trust-remote-code

Citation

@inproceedings{lasby2025reap,
  title={{REAP} the Experts: Why Pruning Prevails for One-Shot {MoE} Compression},
  author={Lasby, Mike and others},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2026},
  url={https://arxiv.org/abs/2510.13999}
}

Links

• —REAP paper: arxiv.org/abs/2510.13999
• —REAP code: github.com/cerebras/reap
• —20% pruned variant: 0xSero/gemma-4-21b-a4b-it-REAP
• —Base model: google/gemma-4-26b-a4b-it