AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant

Gemma 4 31B DECKARD HERETIC Uncensored — NVFP4 SVDQuant

SVDQuant-quantized version of DavidAU/gemma-4-31B-it-The-DECKARD-HERETIC-UNCENSORED-Thinking. Quantized using NVIDIA ModelOpt 0.42.0 with SVDQuant (SVD decomposition + NVFP4) for maximum quality at 4-bit precision. Calibrated natively on NVIDIA B200 (Blackwell SM 12.0) for hardware-accurate FP4 scale factors.

See also: AWQ_FULL variant — same model quantized with AWQ_FULL instead of SVDQuant.

What is SVDQuant?

SVDQuant uses Singular Value Decomposition to separate weight matrices into two components before quantization:

1. 1.Outlier channels — high-magnitude weight channels that cause large quantization error are extracted into a low-rank BF16 residual matrix
2. 2.Cleaned weights — the remaining weights (with outliers removed) are quantized to NVFP4 (E2M1) with dramatically reduced quantization error

This produces higher quality than standard AWQ at the cost of a slightly larger model size (~20.9 GB vs ~20.5 GB) due to the low-rank residual matrices stored in BF16.

Original Weight Matrix W (BF16)
    |
    v
[SVD Decomposition]
    |
    ├── Low-rank residual R (BF16, rank=32) — captures outlier channels
    └── Cleaned weights W' = W - R
            |
            v
        [NVFP4 Quantization] — much lower error without outliers
            |
            v
        W'_quant (FP4 E2M1)

Inference: output = W'_quant @ x + R @ x

Model Details

Quantization Details

Gemma 4 31B DECKARD HERETIC (BF16, ~62 GB)
    |
    v
[NVFP4 SVDQuant on B200]
    - ModelOpt 0.42.0 with NVFP4_SVDQUANT_DEFAULT_CFG
    - Low-rank = 32 (BF16 residual matrices)
    - 2048 calibration samples (CNN DailyMail)
    - Native Blackwell FP4 hardware calibration (SM 12.0)
    - Excluded: vision tower, embed_vision, multi_modal_projector
    - Quantization time: ~69 minutes on B200
    |
    v
Gemma-4-31B-DECKARD-HERETIC-NVFP4-SVDQuant (~20.9 GB)

AWQ_FULL vs SVDQuant Comparison

Deployment

vLLM

vllm serve /path/to/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant \
  --served-model-name deckard-svdquant \
  --quantization modelopt \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --max-model-len 65536 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.70 \
  --trust-remote-code \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4

On the DGX Spark's unified memory keep --gpu-memory-utilization at 0.6-0.7; above ~0.8 the shared CPU+GPU pool page-thrashes. Discrete-VRAM GPUs can run higher.

Docker Compose

services:
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    ports:
      - "8000:8000"
    volumes:
      - ./models:/models
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - VLLM_ATTENTION_BACKEND=FLASHINFER
    command: >
      --model /models/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant
      --served-model-name deckard-svdquant
      --quantization modelopt
      --dtype auto
      --kv-cache-dtype fp8
      --max-model-len 65536
      --max-num-seqs 8
      --gpu-memory-utilization 0.70
      --trust-remote-code
      --enable-chunked-prefill
      --enable-prefix-caching
      --enable-auto-tool-choice
      --tool-call-parser gemma4
      --reasoning-parser gemma4
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

DGX Spark Performance Estimates

Key Deployment Flags

Related Models

• —GitHub repo: AEON-7/Gemma-4-31B-DECKARD-HERETIC-Uncensored-NVFP4 — deployment docs, Docker Compose, cross-model comparison

• —AWQ_FULL variant: AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4 — same base model, AWQ_FULL quantization
• —Gemma 4 MoE NVFP4: AEON-7/Gemma-4-26B-A4B-it-Uncensored-NVFP4 — MoE variant, faster throughput
• —Base model: DavidAU/gemma-4-31B-it-The-DECKARD-HERETIC-UNCENSORED-Thinking

Advanced Techniques

Native B200 Calibration

Quantized on NVIDIA B200 with native FP4 hardware instructions (SM 12.0). The SVDQuant calibration measures actual FP4 rounding behavior on real hardware rather than simulating it, producing more accurate scale factors and SVD decomposition decisions than calibrating on non-FP4 hardware.

SVD Low-Rank Selection

The default low-rank of 32 was used, which provides an excellent balance between quality preservation and model size overhead. Each weight matrix has a 32-column BF16 residual that captures the most important outlier channels, keeping them at full precision while the remaining weights are safely quantized to FP4.

License

This model inherits the Gemma license from the base model.

Legal Disclaimer

THIS MODEL IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND. The authors make no representations regarding accuracy, reliability, or fitness for any purpose. Use at your own risk. By downloading or using this model, you agree that the authors shall not be liable for any claims, damages, or losses arising from its use.

☕ Support the work

If this release has been useful, tips are deeply appreciated — they go directly toward more compute, more models, and more open releases.

<table align="left"> <tr><td align="left"> <strong>₿ Bitcoin (BTC)</strong><br/> <img src="https://raw.githubusercontent.com/AEON-7/AEON-7/main/assets/qr/btc.png" alt="QR" width="200"/><br/> _{<code>bc1q09xmzn00q4z3c5raene0f3pzn9d9pvawfm0py4</code>} </td></tr> <tr><td align="left"> <strong>Ξ Ethereum (ETH)</strong><br/> <img src="https://raw.githubusercontent.com/AEON-7/AEON-7/main/assets/qr/eth.png" alt="QR" width="200"/><br/> _{<code>0x1512667F6D61454ad531d2E45C0a5d1fd82D0500</code>} </td></tr> <tr><td align="left"> <strong>◎ Solana (SOL)</strong><br/> <img src="https://raw.githubusercontent.com/AEON-7/AEON-7/main/assets/qr/sol.png" alt="QR" width="200"/><br/> _{<code>DgQsjHdAnT5PNLQTNpJdpLS3tYGpVcsHQCkpoiAKsw8t</code>} </td></tr> <tr><td align="left"> <strong>ⓜ Monero (XMR)</strong><br/> <img src="https://raw.githubusercontent.com/AEON-7/AEON-7/main/assets/qr/xmr.png" alt="QR" width="200"/><br/> _{<code>836XrSKw4R76vNi3QPJ5Fa9ugcyvE2cWmKSPv3AhpTNNKvqP8v5ba9JRL4Vh7UnFNjDz3E2GXZDVVenu3rkZaNdUFhjAvgd</code>} </td></tr> </table>

Ethereum L2s (Base, Arbitrum, Optimism, Polygon, etc.) and EVM-compatible tokens can be sent to the same Ethereum address.