FadedRedStar/gemma-4-E4B-it-qat-q4_0-heretic-GGUF

🤖 gemma-4-E4B-it-qat-q4_0-heretic — GGUF

This repository hosts GGUF weights and the associated vision/audio projection matrix for gemma-4-E4B-it-qat-q4_0-heretic, quantized from the source floating-point tensors provided by coder3101/gemma-4-E4B-it-qat-q4_0-unquantized-heretic.

🎯 QAT Architecture & Constraints

This model was produced using Quantization Aware Training (QAT). Because the network parameters were exposed to 4-bit constraints natively during fine-tuning, standard post-training formats such as Q4_K_M or Q5_K_M are not compatible and would degrade output quality.

This repository instead provides a single `Q4_0` build, produced with the llama.cpp --pure flag, which matches the numerical profile the model was trained on.

ℹ️ Model Profile & Core Features

Gemma 4 E4B is a Google DeepMind model from the Gemma 4 family, a multimodal (text, image, audio) architecture featuring hybrid sliding-window/global attention, configurable thinking mode, native function calling, and system-role support. This is the Quantization-Aware Training (QAT) variant, whose weights were trained to retain near-bfloat16 quality once quantized to 4-bit — unlike a naively post-training-quantized model, QAT checkpoints are specifically calibrated during training for low-bit deployment. The "E" in E4B denotes "effective" parameters: the model uses Per-Layer Embeddings (PLE) so its effective footprint (4.5B) is much smaller than its total parameter count (8B with embeddings).

The heretic suffix denotes post-processing via [Heretic v1.2.0](https://github.com/p-e-w/heretic) with the Arbitrary-Rank Ablation (ARA) method (row-norm preservation) performed by coder3101, which suppresses refusal behavior while preserving the model's reasoning and multimodal capabilities.

📋 Technical Specifications

🛠️ Heretic Overrides (ARA)

📊 Refusal Bypass Metrics

[!NOTE] The metrics below are self-reported by the original model author (coder3101) and have not been independently reproduced.

🧮 Numerical & Tensor Formats

[!NOTE] Unlike standard Q40 builds that mix in higher-precision tensors for embeddings and select layers, this quant was produced with the `--pure` flag, forcing all eligible tensors to Q40. This most closely mirrors the QAT calibration target and is the recommended pairing for this checkpoint.

📦 Available Model Files

Main model weights | Filename | Quantization | llama.cpp Build | Size | Download | |---|---|---|---|---| | gemma-4-E4B-it-qat-q4_0-heretic-Q4_0_PURE.gguf | Q4_0_PURE | b9851 | 3.96 GB | 📥 Download |

mmproj — vision/audio projector files | Filename | Quantization | Size | Download | |---|---|---|---| | mmproj-gemma-4-E4B-it-qat-q4_0-heretic-Q8_0.gguf | Q8_0 | 534 MB | 📥 Download | | mmproj-gemma-4-E4B-it-qat-q4_0-heretic-BF16.gguf | BF16 | 946 MB | 📥 Download |

🎛️ Component Pairing Guide

• —`Q4_0_PURE`: The only main-weights format provided; required to match the QAT training profile.

mmproj files (optional): multimodal vision projectors. Pass one via the --mmproj flag in llama.cpp to enable image input.

• —`BF16` (Recommended): Highest possible image processing accuracy. While older projectors were small, modern vision towers can hover around 1GB. If you are tight on VRAM, it is completely viable to run this on system RAM (CPU) with a minimal performance penalty, saving your precious GPU space for the main model layers.
• —`Q8_0`: Cuts the projector file size and memory footprint in half (~500MB for larger 1GB files). Use this if you prefer to keep the vision tower hosted entirely on your GPU but need to claw back some VRAM to avoid Out-Of-Memory (OOM) crashes.

⚡ Deployment & Execution Commands

[!IMPORTANT] The vision projector (--mmproj) must be supplied at runtime whenever image inputs are used. Omitting it disables multimodal capability entirely.

[!NOTE] Google recommends the following sampling configuration for best results: temperature=1.0, top_p=0.95, top_k=64.

[!NOTE] Thinking is enabled by including the <|think|> token at the start of the system prompt; remove it to disable reasoning output.

llama.cpp CLI (with image)

./llama-cli \
  -m gemma-4-E4B-it-qat-q4_0-heretic-Q4_0_PURE.gguf \
  --mmproj mmproj-gemma-4-E4B-it-qat-q4_0-heretic-BF16.gguf \
  -c 8192 \
  -ngl 99 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 64 \
  --image "path/to/image.jpg" \
  -p "<start_of_turn>user\nDescribe what you see in this image.<end_of_turn>\n<start_of_turn>model\n"

OpenAI-Compatible API Server

./llama-server \
  --host 0.0.0.0 \
  --port 8080 \
  -m gemma-4-E4B-it-qat-q4_0-heretic-Q4_0_PURE.gguf \
  --mmproj mmproj-gemma-4-E4B-it-qat-q4_0-heretic-BF16.gguf \
  -c 16384 \
  -ngl 99 \
  --flash-attn

💬 Chat Templates & Prompt Design (Gemma)

<start_of_turn>system
You are a helpful assistant.<end_of_turn>
<start_of_turn>user
Your question, image, or audio payload here.<end_of_turn>
<start_of_turn>model

⚠️ Safety & Operational Notes

• —This model is abliterated and will generate content that standard aligned models refuse. Use responsibly and in compliance with applicable laws.
• —Audio input is limited to 30 seconds; video is processed as frames at up to 60 seconds (1 fps).
• —Place image content before text and audio content after text in the prompt for best results.
• —In multi-turn conversations, do not carry prior turns' thinking content forward — only the final response should appear in history.
• —Vision tensors are kept at BF16 or Q8_0 depending on the mmproj variant chosen, to preserve spatial feature quality.
• —The --pure Q4_0 quantization applies uniform 4-bit precision across all eligible tensors rather than mixing precisions, which most closely matches this checkpoint's QAT calibration target.