AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4

Gemma 4 E4B Uncensored NVFP4

EAGLE speculative decoding drafter for Gemma 4 26B-A4B Uncensored (TrevorJS) — a 42-layer E4B (EAGLE for Blackwell) model quantized to NVFP4 AWQ using NVIDIA ModelOpt 0.42.0.

Designed for EAGLE-based speculative decoding on NVIDIA DGX Spark (GB10, SM 12.1) and other Blackwell-architecture GPUs.

[GitHub Repo](https://github.com/AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4) — patches, deployment configs, Dockerfile

Model Details

Quick Start

Prerequisites

1. 1.Target model — Any NVFP4-quantized Gemma 4 26B MoE (e.g., AEON-7/Gemma-4-26B-A4B-it-Uncensored-NVFP4)
2. 2.This drafter model — Download below
3. 3.Three vLLM patches — Required for Gemma 4 speculative decoding (see Required Patches)
4. 4.Pre-built container — ghcr.io/aeon-7/vllm-spark-gemma4-nvfp4-awq:latest

1. Download both models

pip install -U huggingface-hub

# Target model (26B MoE)
huggingface-cli download AEON-7/Gemma-4-26B-A4B-it-Uncensored-NVFP4 \
  --local-dir ~/models/trevorjs-26b

# This drafter model (E4B)
huggingface-cli download AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4 \
  --local-dir ~/models/e4b-drafter

2. Get the patched vLLM files

Three patches to vLLM 0.19.1 are required for Gemma 4 speculative decoding. Download from the DECKARD 31B GitHub repo:

for f in eagle_patched.py serving_chat_patched.py modelopt_patched.py; do
  curl -LO https://raw.githubusercontent.com/AEON-7/Gemma-4-31B-DECKARD-HERETIC-Uncensored-NVFP4/main/$f
done

3. Launch with Docker Compose

services:
  vllm:
    image: ghcr.io/aeon-7/vllm-spark-gemma4-nvfp4-awq:latest
    container_name: vllm-trevorjs-26b-spec
    restart: unless-stopped
    network_mode: host
    volumes:
      - ~/models/trevorjs-26b:/models/target
      - ~/models/e4b-drafter:/models/e4b-drafter
      - ./modelopt_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/modelopt.py
      - ./serving_chat_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py
      - ./eagle_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/v1/spec_decode/eagle.py
    environment:
      - VLLM_TEST_FORCE_FP8_MARLIN=1
      - VLLM_MARLIN_USE_ATOMIC_ADD=1
      - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
      - VLLM_USE_FLASHINFER_MOE_FP4=1
      - TORCH_MATMUL_PRECISION=high
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
    command:
      - bash
      - -c
      - |
        exec vllm serve /models/target \
          --served-model-name trevorjs-26b \
          --quantization modelopt \
          --dtype auto \
          --kv-cache-dtype fp8 \
          --tensor-parallel-size 1 \
          --max-model-len 131072 \
          --max-num-seqs 4 \
          --gpu-memory-utilization 0.65 \
          --trust-remote-code \
          --host 0.0.0.0 --port 8000 \
          --enable-chunked-prefill \
          --enable-prefix-caching \
          --enable-auto-tool-choice \
          --tool-call-parser gemma4 \
          --reasoning-parser gemma4 \
          --speculative-config '{"method":"draft_model","model":"/models/e4b-drafter","num_speculative_tokens":5,"quantization":"modelopt"}'
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

On the DGX Spark's unified memory keep --gpu-memory-utilization at 0.6-0.7; above ~0.8 the shared CPU+GPU pool page-thrashes and stalls the box, and a spec-decode drafter's verify buffers are not counted by the fraction so leave headroom (0.65 here). Discrete-VRAM GPUs can run higher.

Speculative Config Parameters

Required vLLM Patches

Speculative decoding with Gemma 4 requires three patches to vLLM 0.19.1. Without these, the server will crash on startup.

Patch 1: eagle_patched.py — Gemma 4 spec decode support

File: vllm/v1/spec_decode/eagle.py

Three fixes are needed:

1a. Remove multimodal guard

vLLM 0.19.1 calls _raise_if_multimodal() which blocks ALL multimodal targets from speculative decoding, even when the drafter is text-only. Remove this call — the downstream code already handles text-only drafters with multimodal targets correctly.

# In initialize() method — REMOVE this line:
# self._raise_if_multimodal()

1b. Add Gemma4 to model whitelist

Gemma 4 uses image_token_id (258880) but NOT image_token_index. The spec decode framework needs an explicit mapping:

# In the model whitelist check, add Gemma4:
if self.get_model_name(target_model) in [
    "Qwen2_5_VLForConditionalGeneration",
    # ... existing models ...
    "Gemma4ForConditionalGeneration",  # ADD THIS
]:
    self.model.config.image_token_index = target_model.config.image_token_id

1c. Multi-group KV cache support

Gemma 4 uses heterogeneous attention: head_dim=256 for sliding-window layers and head_dim=512 for global attention layers. This creates two distinct KV cache groups. The spec decode framework assumes a single group.

Fix validate_same_kv_cache_group to log a warning instead of asserting, and rewrite initialize_attn_backend to key attention groups by (backend_class, kv_cache_group_id) instead of just backend_class, mapping each draft layer to its correct KV cache group.

Patch 2: serving_chat_patched.py — Non-streaming reasoning parser

File: vllm/entrypoints/openai/chat_completion/serving.py

Gemma 4's reasoning parser uses <|channel> (token 100) and <channel|> (token 101) delimiters. With skip_special_tokens=True (the default for non-streaming), these are stripped, causing extract_reasoning() to return None — thinking content lands in the content field.

The fix re-decodes from raw token_ids with skip_special_tokens=False when text-based extraction fails:

if reasoning is None and token_ids and hasattr(reasoning_parser, 'start_token_id'):
    token_ids_list = list(token_ids)
    if reasoning_parser.start_token_id in token_ids_list:
        full_text = reasoning_parser.model_tokenizer.decode(
            token_ids_list, skip_special_tokens=False
        )
        reasoning, content = reasoning_parser.extract_reasoning(full_text, request=request)
        if content:
            _tok = reasoning_parser.model_tokenizer
            content = _tok.decode(_tok.encode(content), skip_special_tokens=True)

Patch 3: modelopt_patched.py — NVFP4 AWQ support

File: vllm/model_executor/layers/quantization/modelopt.py

Three fixes:

1. 1.FP8 NaN scrubbing — ModelOpt 0.42.0 produces ~60 FP8 NaN values (0x7F/0xFF) in weight_scale tensors. Scrubs to zero at load time.
2. 2.NVFP4_AWQ quant_algo — Registers NVFP4_AWQ (upstream only handles NVFP4).
3. 3.AWQ pre_quant_scale — Loads and applies per-channel pre_quant_scale tensors for AWQ weight redistribution.

Applying the Patches

Mount as volume binds in Docker Compose (shown above), or copy manually:

VLLM_PATH=$(python3 -c "import vllm; print(vllm.__path__[0])")
cp eagle_patched.py    $VLLM_PATH/v1/spec_decode/eagle.py
cp serving_chat_patched.py $VLLM_PATH/entrypoints/openai/chat_completion/serving.py
cp modelopt_patched.py $VLLM_PATH/model_executor/layers/quantization/modelopt.py

Heterogeneous Attention Architecture

This E4B drafter mirrors the Gemma 4 heterogeneous attention design:

• —35 sliding-window layers — head_dim=256, window of 512 tokens, default RoPE (theta=10000)
• —7 full-attention layers — head_dim=512, global attention, proportional RoPE (theta=1M, partialrotaryfactor=0.25)

This creates two distinct KV cache groups within the drafter, handled by the multi-group KV cache fix in eagle_patched.py.

Cross-Model Drafter Compatibility

This drafter was derived from the TrevorJS uncensored fine-tune. It can also be used with other Gemma 4 targets that share the same vocabulary (262K tokens):

Acceptance rate will be highest with the matching base model and lower with mismatched fine-tunes, but all combinations will function correctly.

Related Models

Hardware Requirements

• —Target + Drafter combined: ~26 GB (16 GB target + 9.6 GB drafter)
• —Recommended: NVIDIA DGX Spark (128 GB unified memory) or any GPU with >= 40 GB VRAM
• —Required: Blackwell architecture (SM 10.0+) for native FP4

License

This model inherits the Gemma license from Google.

☕ Support the work

If this release has been useful, tips are deeply appreciated — they go directly toward more compute, more models, and more open releases.

<table align="left"> <tr><td align="left"> <strong>₿ Bitcoin (BTC)</strong><br/> <img src="https://raw.githubusercontent.com/AEON-7/AEON-7/main/assets/qr/btc.png" alt="QR" width="200"/><br/> _{<code>bc1q09xmzn00q4z3c5raene0f3pzn9d9pvawfm0py4</code>} </td></tr> <tr><td align="left"> <strong>Ξ Ethereum (ETH)</strong><br/> <img src="https://raw.githubusercontent.com/AEON-7/AEON-7/main/assets/qr/eth.png" alt="QR" width="200"/><br/> _{<code>0x1512667F6D61454ad531d2E45C0a5d1fd82D0500</code>} </td></tr> <tr><td align="left"> <strong>◎ Solana (SOL)</strong><br/> <img src="https://raw.githubusercontent.com/AEON-7/AEON-7/main/assets/qr/sol.png" alt="QR" width="200"/><br/> _{<code>DgQsjHdAnT5PNLQTNpJdpLS3tYGpVcsHQCkpoiAKsw8t</code>} </td></tr> <tr><td align="left"> <strong>ⓜ Monero (XMR)</strong><br/> <img src="https://raw.githubusercontent.com/AEON-7/AEON-7/main/assets/qr/xmr.png" alt="QR" width="200"/><br/> _{<code>836XrSKw4R76vNi3QPJ5Fa9ugcyvE2cWmKSPv3AhpTNNKvqP8v5ba9JRL4Vh7UnFNjDz3E2GXZDVVenu3rkZaNdUFhjAvgd</code>} </td></tr> </table>

Ethereum L2s (Base, Arbitrum, Optimism, Polygon, etc.) and EVM-compatible tokens can be sent to the same Ethereum address.