AEON-7/supergemma4-26b-abliterated-multimodal-nvfp4

SuperGemma4 26B Abliterated Multimodal — NVFP4

NVFP4-quantized version of Jiunsong/supergemma4-26b-abliterated-multimodal — an abliterated (uncensored) Gemma 4 26B Mixture-of-Experts multimodal model with thinking/reasoning capabilities.

Quantized using NVIDIA ModelOpt 0.43 (main) with NVFP4_DEFAULT_CFG on a native Blackwell GPU. Vision encoder preserved in full BF16. Peak aggregate throughput: 1,890 tok/s @ 256 concurrent on DGX Spark (GB10).

Verified end-to-end: calibrated → exported → served on Spark → benchmarked 1-256 concurrency.

⚠️ IMPORTANT REQUIREMENTS — READ THIS FIRST

This model has non-obvious serving requirements because its per-expert-decomposed NVFP4 scale format needs specific plugin handling. Deviating from these will produce garbage output or crashes. Details below — each requirement is backed by hours of debugging.

🔴 MUST-DO requirements

✅ Verified-working config (use this verbatim)

services:
  vllm:
    image: ghcr.io/aeon-7/vllm-spark-gemma4-nvfp4-awq:latest   # NOT the non-awq variant
    environment:
      - VLLM_TEST_FORCE_FP8_MARLIN=1     # required for MoE
      - VLLM_MARLIN_USE_ATOMIC_ADD=1
      - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
      - TORCH_MATMUL_PRECISION=high
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
      - NVIDIA_FORWARD_COMPAT=1
    volumes:
      # model + 2 patches ONLY — do not mount modelopt_patched.py (it's baked in)
      - ./model:/models/supergemma4
      - ./gemma4_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/gemma4.py
      - ./serving_chat_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py
    command: >
      vllm serve /models/supergemma4
      --quantization modelopt
      --kv-cache-dtype fp8_e4m3
      --tensor-parallel-size 1
      --max-model-len 65536
      --max-num-seqs 4
      --gpu-memory-utilization 0.70
      --trust-remote-code
      --host 0.0.0.0 --port 8000
      --enable-chunked-prefill
      --enable-prefix-caching
      --enable-auto-tool-choice
      --tool-call-parser gemma4
      --reasoning-parser gemma4

❌ Known FAILURE modes (things that DON'T work)

🐛 If you see gibberish output after following all of the above

1. 1.Verify the image: docker inspect <container> | grep Image should show vllm-spark-gemma4-nvfp4-awq
2. 2.Verify mounts: docker inspect <container> --format '{{json .Mounts}}' should show exactly 3 mounts (model + 2 patches)
3. 3.Verify backend selection in logs:
4. 4.Using NvFp4LinearBackend.FLASHINFER_CUTLASS for NVFP4 GEMM ✅
5. 5.Using 'MARLIN' NvFp4 MoE backend ✅
6. 6.Test with raw chat: {"messages": [{"role": "user", "content": "Capital of France? One sentence."}]} — should return "The capital of France is Paris.". If not, check the container logs for crashes or UNEXPECTED tensor warnings at load time.

Performance Benchmarks

NVIDIA DGX Spark (GB10, SM 12.0, 128 GB unified memory) — vLLM 0.19.1rc1.dev110+gb55d830ec, FP8 E4M3 KV cache, native FlashInfer CUTLASS linear + Marlin MoE backend, --gpu-memory-utilization 0.85.

1. Single-Stream Performance (README-spec config)

`--max-num-seqs 4`, `--max-model-len 65536`, `--gpu-memory-utilization 0.85`. Best for interactive chat, agentic UX, single-user serving. All measurements greedy sampling (temp=0) unless noted.

Decode rate (10 trials, 200 tokens output)

Extremely stable — ±0.5 tok/s variance across 10 trials.

TTFT by prompt length

Time from request to first token, across 5 trials each:

Even 465-token prompts give sub-50ms TTFT — fixed kernel-launch overhead dominates over prefill for anything < ~500 tokens.

Decode rate by output length

Longer outputs are slightly slower due to growing KV cache:

*Short because model hit EOS naturally before 1000 tokens.

Sampling: Greedy vs Stochastic

Temperature has negligible performance impact:

Long-prompt prefill (RAG / document workloads)

Prefill throughput scales impressively with length — MoE's sparse compute is the perfect shape for prefill:

Decode only drops 7% at 32K context — excellent KV-cache bandwidth behavior. Prefill peaks around 194K tok/s at 32K prompt length.

Summary

This matches and exceeds the original v6 validation (52.6 tok/s / 54 ms TTFT).

2. Concurrent-Session Performance (max-throughput config)

`--max-num-seqs 256`, `--max-model-len 2048`, `--max-num-batched-tokens 16384`, `--gpu-memory-utilization 0.85`. Best for agent fleets, multi-user serving, batch inference. 3 trials per level with median reported. Mixed prompts (code, math, QA, creative), 200 token output, temp=0.7, SSE streaming.

Throughput scaling (N concurrent clients, 200-tok output)

Zero errors across 1,200+ requests in the full test. Aggregate throughput scales nearly linearly up to 128 concurrent, with diminishing returns at 256 as scheduling and KV-cache contention dominate.

Note: single-stream here is 35.9 tok/s (vs 51.1 in README config) because max-num-seqs=256 forces allocation of 50+ CUDA graph sizes and different scheduling heuristics that optimize for batched throughput over single-stream latency. Use README config for chat; use this config for fleets.

TTFT-only scaling (prefill + first token, 1-token output)

Measures how much queue contention affects time-to-first-token — critical for agent UX:

TTFT stays sub-200ms up through 64 concurrent — smooth UX for small agent fleets. Above 128 concurrent TTFT doubles per level as requests queue for scheduler capacity.

Concurrent with 1K-token prompts (RAG-style workload)

50-token output with 1,024-token prompts — simulates agents doing document QA or retrieval-augmented responses:

Long-prompt concurrent workloads scale as well as short-prompt ones (prefill is very fast on MoE with 194K tok/s peak throughput).

Summary

Key Performance Metrics

Scaling Efficiency

Aggregate throughput scales 50x from 1 to 256 concurrent requests — excellent batching efficiency from the MoE architecture. Per-request throughput degrades gracefully from 37.8 tok/s (1-req) to 9.3 tok/s (256-req), still usable for agent workloads with many short-lived subagents.

Why MoE is Fast on DGX Spark

GB10's 273 GB/s memory bandwidth is the bottleneck for LLM decode. MoE dramatically reduces per-token bandwidth demand:

Key Specs

Model Details

Pre-Built Container Image

A pre-built vLLM container compiled for NVIDIA DGX Spark (GB10, SM 12.1) is available with all required patches pre-applied:

docker pull ghcr.io/aeon-7/vllm-spark-gemma4-nvfp4-awq:latest

Image contents:

• —vLLM 0.19.1rc1 compiled for SM 12.1 (Blackwell GB10)
• —PyTorch 2.12.0 + CUDA 13.0
• —transformers 5.5.0 + FlashInfer 0.6.7
• —Patched gemma4.py — extends expert_params_mapping to the modelopt suffix set (weight, weight_scale, weight_scale_2, input_scale)
• —Patched serving.py — fixes non-streaming reasoning parser for Gemma 4
• —Patched modelopt.py — handles the per-expert-decomposed NVFP4 scale format
• —Built from eugr/spark-vllm-docker with --tf5 flag

Critical: Use the -awq variant of the image. The non--awq image does not include the baked-in modelopt scale-handling patches required for this model's per-expert NVFP4 format.

Container on GHCR

Quick Start

1. Pull the container

docker pull ghcr.io/aeon-7/vllm-spark-gemma4-nvfp4-awq:latest

2. Download the model

pip install -U huggingface-hub hf_transfer

HF_HUB_ENABLE_HF_TRANSFER=1 \
  hf download AEON-7/supergemma4-26b-abliterated-multimodal-nvfp4 \
  --local-dir ~/models/supergemma4-26b

3. Get the patches

Only two patch files need to be mounted — modelopt.py is baked into the -awq image:

for f in gemma4_patched.py serving_chat_patched.py; do
  curl -LO https://raw.githubusercontent.com/AEON-7/supergemma4-26b-abliterated-multimodal-nvfp4/main/$f
done

4. Launch with Docker Compose

Save as docker-compose.yml:

services:
  vllm:
    image: ghcr.io/aeon-7/vllm-spark-gemma4-nvfp4-awq:latest
    container_name: vllm-supergemma4-26b
    restart: unless-stopped
    network_mode: host
    volumes:
      - ~/models/supergemma4-26b:/models/supergemma4
      - ./gemma4_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/gemma4.py
      - ./serving_chat_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py
    environment:
      # Force Marlin MoE path — native FlashInfer MoE variants reject 704-per-expert intermediate
      - VLLM_TEST_FORCE_FP8_MARLIN=1
      - VLLM_MARLIN_USE_ATOMIC_ADD=1
      - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
      - TORCH_MATMUL_PRECISION=high
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
      - NVIDIA_FORWARD_COMPAT=1
    command:
      - bash
      - -c
      - |
        exec vllm serve /models/supergemma4 \
          --served-model-name supergemma4-26b \
          --quantization modelopt \
          --dtype auto \
          --kv-cache-dtype fp8_e4m3 \
          --tensor-parallel-size 1 \
          --max-model-len 65536 \
          --max-num-seqs 4 \
          --gpu-memory-utilization 0.70 \
          --trust-remote-code \
          --host 0.0.0.0 --port 8000 \
          --enable-chunked-prefill \
          --enable-prefix-caching \
          --enable-auto-tool-choice \
          --tool-call-parser gemma4 \
          --reasoning-parser gemma4
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Then:

docker compose up -d

Startup takes ~4-5 minutes (weight load + torch.compile + CUDA graph capture + FP4 GEMM autotuning).

Workload-tuned configs

5. Test

# Text
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "supergemma4-26b",
    "messages": [{"role": "user", "content": "Explain quantum entanglement simply."}],
    "max_tokens": 300
  }'

# Vision
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "supergemma4-26b",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/PNG_transparency_demonstration_1.png/280px-PNG_transparency_demonstration_1.png"}},
        {"type": "text", "text": "Describe what you see."}
      ]
    }],
    "max_tokens": 300
  }'

The API is fully OpenAI-compatible — use with any OpenAI SDK, LangChain, LiteLLM, Open WebUI at http://<your-ip>:8000/v1.

Key Deployment Flags

Quantization Details

Why plain NVFP4 instead of NVFP4_AWQ?

Earlier experiments used NVFP4_AWQ_FULL_CFG (AWQ with exhaustive alpha grid search) but ran into a deployment-stack limitation: vLLM's ModelOptNvFp4FusedMoE does not support per-expert pre_quant_scale. On MoE models, AWQ calibration computes a per-expert scaling factor that can't be consumed by the MoE kernel path — any AWQ work on experts is wasted at serve time.

Switching to plain NVFP4 (algorithm=max):

• —Cuts calibration time from ~2.5h to ~12 min (no alpha search phase)
• —Produces a checkpoint vLLM's FusedMoE loads natively without tensor surgery
• —Quality hit is negligible since the AWQ benefit on MoE experts was already unavailable at inference time

Attention and dense shared MLP layers still benefit from NVFP4's per-block scaling. Router weights stay in BF16 (routing quality is critical for MoE accuracy and experts are cheap to leave un-quantized there).

Applied modelopt patches

Two upstream PR fixes applied locally (pending review as of this writing):

• —PR #1264 — preprocess_linear_fusion non-scalar amax fix
• —PR #1265 — get_activation_scaling_factor zero-amax handling

Both are blockers for anyone quantizing per-expert-decomposed MoEs in NVFP4 with modelopt 0.42 or 0.43. The -awq container image includes these patches in its baked modelopt.py — do not override with a stock version.

fast-moe adaptive batched calibration

Calibration uses `modelopt-fast-moe` — adaptive VRAM-probed batching that fixes the Python-dispatch bottleneck when quantizing MoE models (the naive for ids in calib_data: model(ids) loop leaves GPUs at 25-30% utilization).

End-to-end calibration wall-clock:

NVFP4 Weight Format

Each quantized layer stores:

• —weight (uint8) — packed FP4 E2M1 pairs (16-element blocks)
• —weight_scale (float8_e4m3fn) — per-block scale (1 per 16 elements)
• —weight_scale_2 (float32) — per-tensor global scale (stored as modelopt reciprocal convention)
• —input_scale (float32) — static activation scale from calibration

Quality Validation

Greedy-sampled responses (temperature=0.0):

Speculative Decoding (DFlash — Coming Soon)

A DFlash block-diffusion drafter paired with this model is in training. DFlash can provide 2-3× additional throughput over the numbers above by predicting multi-token blocks in a single draft forward pass. Will be published as a separate drafter repo once training completes.

Dense (31B) vs MoE (26B) Comparison

Hardware Requirements

Native FP4 hardware (Blackwell architecture) is required — will not run on Ampere or Ada GPUs.

Related Projects

Models

Infrastructure

Disclaimer

THIS IS AN UNCENSORED MODEL. By downloading, accessing, or using this model, you expressly acknowledge that you assume full and sole responsibility for all outputs generated, all actions taken based on outputs, and compliance with applicable laws. The authors are not responsible for any harmful, illegal, or objectionable content produced by the model. These tools serve legitimate purposes including security research, red-teaming, content analysis, and creative work. Implement safeguards appropriate to your use case and jurisdiction.

License

This model inherits the Gemma license from Google.

Credits

Quantized by AEON-7 on NVIDIA Blackwell hardware. Built and validated with AI-engineering assistance from Anthropic.

Shout-out to eugr/spark-vllm-docker for the DGX Spark-optimized vLLM build, NVIDIA for TensorRT-Model-Optimizer, and the z-lab / ModelOpt teams for DFlash.

☕ Support the work

If this release has been useful, tips are deeply appreciated — they go directly toward more compute, more models, and more open releases.

<table align="left"> <tr><td align="left"> <strong>₿ Bitcoin (BTC)</strong><br/> <img src="https://raw.githubusercontent.com/AEON-7/AEON-7/main/assets/qr/btc.png" alt="QR" width="200"/><br/> _{<code>bc1q09xmzn00q4z3c5raene0f3pzn9d9pvawfm0py4</code>} </td></tr> <tr><td align="left"> <strong>Ξ Ethereum (ETH)</strong><br/> <img src="https://raw.githubusercontent.com/AEON-7/AEON-7/main/assets/qr/eth.png" alt="QR" width="200"/><br/> _{<code>0x1512667F6D61454ad531d2E45C0a5d1fd82D0500</code>} </td></tr> <tr><td align="left"> <strong>◎ Solana (SOL)</strong><br/> <img src="https://raw.githubusercontent.com/AEON-7/AEON-7/main/assets/qr/sol.png" alt="QR" width="200"/><br/> _{<code>DgQsjHdAnT5PNLQTNpJdpLS3tYGpVcsHQCkpoiAKsw8t</code>} </td></tr> <tr><td align="left"> <strong>ⓜ Monero (XMR)</strong><br/> <img src="https://raw.githubusercontent.com/AEON-7/AEON-7/main/assets/qr/xmr.png" alt="QR" width="200"/><br/> _{<code>836XrSKw4R76vNi3QPJ5Fa9ugcyvE2cWmKSPv3AhpTNNKvqP8v5ba9JRL4Vh7UnFNjDz3E2GXZDVVenu3rkZaNdUFhjAvgd</code>} </td></tr> </table>

Ethereum L2s (Base, Arbitrum, Optimism, Polygon, etc.) and EVM-compatible tokens can be sent to the same Ethereum address.