Modelpublic

sakamakismile/Huihui-gemma-4-31B-it-abliterated-v2-NVFP4

sourceHugging Facegemmaupdated 6mo agoView on Hugging Face

2likes164downloads

Model Card

Huihui-gemma-4-31B-it-abliterated-v2-NVFP4

NVFP4 quantized version of huihui-ai/Huihui-gemma-4-31B-it-abliterated-v2 — an abliterated (uncensored) Gemma 4 31B dense model with multimodal capability.

59 GB → 20.5 GB. Single NVIDIA Blackwell GPU.

Why This Model

Gemma 4 31B Dense uses all 31B parameters on every token — unlike the 26B-A4B MoE variant (3.8B active). This means:

—Deeper reasoning per token at the cost of speed
—No routing noise — deterministic compute path
—262K context with multimodal (vision) support
—Abliterated — no refusals for local agent workflows

Key Specs


Base model	huihui-ai/Huihui-gemma-4-31B-it-abliterated-v2
Architecture	Gemma 4 Dense — 31B parameters, 60 layers
Quantization	NVFP4 W4A4 (weights FP4, activations FP4, scales FP8)
Format	`compressed-tensors` (native vLLM support)
Tool	vllm-project/llm-compressor (main)
Calibration	32 samples, neuralmagic/calibration, seq_len=2048
Size	20.5 GB
Max context	262,144 tokens
Requires	NVIDIA Blackwell GPU (SM 120)

Quickstart

Standard (vLLM)

bash

vllm serve Lna-Lab/Huihui-gemma-4-31B-it-abliterated-v2-NVFP4 \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90

With reasoning + tool calling

bash

vllm serve Lna-Lab/Huihui-gemma-4-31B-it-abliterated-v2-NVFP4 \
    --max-model-len 32768 \
    --reasoning-parser gemma4 \
    --tool-call-parser gemma4 \
    --enable-auto-tool-choice

Docker

bash

docker run --gpus '"device=0"' -p 8016:8016 \
    -v /path/to/model:/models/current:ro \
    --shm-size 16gb \
    vllm/vllm-openai:cu130-nightly \
    vllm serve /models/current --port 8016 --max-model-len 32768

Benchmark

Single NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM).

Test	Speed	Tokens	Result
English (CAP theorem)	51.4 tok/s	256	PASS
Code (async fetcher)	51.4 tok/s	512	PASS
Math (Bayes' theorem)	51.3 tok/s	512	PASS
Container burst (×3)	51.6 tok/s	256	PASS — stable

Sustained: ~51 tok/s (single GPU, CUDA Graph PIECEWISE).

Slower than Gemma 4 26B-A4B MoE (~160 tok/s) because all 31B parameters are active per token vs. 3.8B for MoE. Trade-off: deeper reasoning per token.

Quantization Details

Recipe

python

from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=["re:.*vision.*", "re:.*audio.*", "lm_head", "re:.*embed.*"],
)

Following RedHatAI/gemma-4-31B-it-NVFP4 proven recipe.

Calibration

—Dataset: neuralmagic/calibration (LLM split)
—Samples: 32
—Max sequence length: 2048

Reproduction

python

from transformers import Gemma4ForConditionalGeneration, AutoProcessor
from datasets import load_dataset
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
import torch

MODEL_ID = "huihui-ai/Huihui-gemma-4-31B-it-abliterated-v2"
model = Gemma4ForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto", trust_remote_code=True)
processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)

recipe = QuantizationModifier(
    targets="Linear", scheme="NVFP4",
    ignore=["re:.*vision.*", "re:.*audio.*", "lm_head", "re:.*embed.*"],
)

ds = load_dataset("neuralmagic/calibration", name="LLM", split="train[:32]")
def preprocess(example):
    messages = [{"role": m["role"], "content": [{"type": "text", "text": m["content"]}]}
                for m in example["messages"]]
    return processor.apply_chat_template(messages, return_tensors="pt", padding=False,
        truncation=True, max_length=2048, tokenize=True, add_special_tokens=False,
        return_dict=True, add_generation_prompt=False)
ds = ds.map(preprocess, batched=False, remove_columns=ds.column_names)

def data_collator(batch):
    assert len(batch) == 1
    return {k: (torch.tensor(v) if k != "pixel_values"
                else torch.tensor(v, dtype=torch.bfloat16).squeeze(0))
            for k, v in batch[0].items()}

oneshot(model=model, recipe=recipe, dataset=ds,
        max_seq_length=2048, num_calibration_samples=32, data_collator=data_collator)

model.save_pretrained("output-NVFP4", save_compressed=True)
processor.save_pretrained("output-NVFP4")

Environment

Package	Version
torch	2.11.0+cu130
transformers	5.5.4
llmcompressor	0.1.dev (main @ 3084520)
compressed-tensors	0.15.1a20260414
CUDA	13.0

Requirements

—GPU: NVIDIA Blackwell (SM 120)
—VRAM: 21 GB minimum (model only), ~92 GB for 128K context
—Software: vLLM nightly (cu130)

Notes

—Abliterated (uncensored). Use responsibly.
—Vision tower preserved in BF16 — multimodal works.
—NVFP4 is Blackwell-specific. Will not work on Ampere/Hopper.
—Dense 31B is slower than MoE 26B-A4B (~51 vs ~160 tok/s) but reasons deeper per token.

Credits

—Base model: huihui-ai (abliteration)
—Original model: Google DeepMind (Gemma 4)
—Quantization recipe: RedHatAI (proven path)
—Quantization tool: llm-compressor

Support the Base Model Author

—Ko-fi: https://ko-fi.com/huihuiai
—Bitcoin: bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge