sakamakismile/Huihui-gemma-4-26B-A4B-it-abliterated-NVFP4

Huihui-gemma-4-26B-A4B-it-abliterated-NVFP4

NVFP4 quantized version of huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated — an abliterated (uncensored) variant of Google's Gemma 4 26B-A4B Mixture-of-Experts model.

49 GB → 16.5 GB — runs on a single NVIDIA Blackwell GPU at ~160 tok/s.

### Known Issue: Japanese / Non-English Long-Form Generation NVFP4 quantization of the 26B (128-expert) model causes intermittent repetition collapse on long Japanese text generation. The model may produce degenerate output like get(get) get(get)... when generating 500+ tokens in Japanese. English tasks (code, math, reasoning) are unaffected. This appears to be inherent to the 128-expert MoE architecture under FP4 quantization — with fewer experts, quantization noise can corrupt the specific expert combinations needed for non-English generation. For multilingual / Japanese workloads, we recommend the 48B (256-expert) variant instead: sakamakismile/Huihui4-48B-A4B-abliterated-NVFP4 — same inference speed, stable across all languages.

Key Specs

Quickstart

vLLM (recommended)

vllm serve Lna-Lab/Huihui-gemma-4-26B-A4B-it-abliterated-NVFP4 \
    --max-model-len 8192

No --quantization flag needed — vLLM auto-detects compressed-tensors format.

Docker

docker run --gpus '"device=0"' -p 8016:8016 \
    -v /path/to/model:/models/current:ro \
    --shm-size 16gb \
    vllm/vllm-openai:cu130-nightly \
    vllm serve /models/current --port 8016 --max-model-len 8192

Python (vLLM)

from vllm import LLM, SamplingParams

llm = LLM(
    model="Lna-Lab/Huihui-gemma-4-26B-A4B-it-abliterated-NVFP4",
    max_model_len=8192,
    gpu_memory_utilization=0.85,
)

output = llm.generate(
    ["Explain quantum entanglement in simple terms."],
    SamplingParams(max_tokens=256, temperature=0.7),
)
print(output[0].outputs[0].text)

Benchmark

Tested on a single NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM), vLLM 0.19.1+, maxmodellen=8192, temperature=0.0.

Sustained throughput: ~150–160 tok/s (post-warmup, single GPU).

VRAM Usage

Comparison with 48B (256-expert) variant

Quantization Details

Recipe

default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: [lm_head, 're:.*embed.*', 're:.*router', 're:.*vision_tower.*']
      scheme: NVFP4

What's quantized, what's not

• —Quantized (NVFP4): All Linear layers in the language model, including MoE expert layers
• —Kept in BF16: lm_head, all embedding layers, MoE routers, entire vision tower

Calibration

• —Dataset: neuralmagic/calibration (LLM split)
• —Samples: 20
• —Max sequence length: 8192
• —MoE expert calibration handled automatically by llm-compressor's SequentialGemma4TextExperts

Reproduction

from datasets import load_dataset
from transformers import AutoProcessor, Gemma4ForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model = Gemma4ForConditionalGeneration.from_pretrained(
    "huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated", dtype="auto"
)
processor = AutoProcessor.from_pretrained(
    "huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated"
)

recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=["lm_head", "re:.*embed.*", "re:.*router", "re:.*vision_tower.*"],
)

ds = load_dataset("neuralmagic/calibration", name="LLM", split="train[:20]")

def preprocess_function(example):
    messages = [
        {"role": m["role"], "content": [{"type": "text", "text": m["content"]}]}
        for m in example["messages"]
    ]
    return processor.apply_chat_template(
        messages, return_tensors="pt", padding=False, truncation=True,
        max_length=8192, tokenize=True, add_special_tokens=False,
        return_dict=True, add_generation_prompt=False,
    )

ds = ds.map(preprocess_function, batched=False, remove_columns=ds.column_names)

import torch
def data_collator(batch):
    assert len(batch) == 1
    return {
        key: (torch.tensor(value) if key != "pixel_values"
              else torch.tensor(value, dtype=torch.bfloat16).squeeze(0))
        for key, value in batch[0].items()
    }

oneshot(
    model=model, recipe=recipe, dataset=ds,
    max_seq_length=8192, num_calibration_samples=20,
    data_collator=data_collator,
)

model.save_pretrained("output-NVFP4", save_compressed=True)
processor.save_pretrained("output-NVFP4")

Environment

Requirements

• —GPU: NVIDIA Blackwell (RTX 5090, RTX PRO 6000, B200, etc.) — NVFP4 requires SM 120
• —VRAM: ~16 GB minimum
• —Software: vLLM nightly (cu130 build), or any framework supporting compressed-tensors NVFP4

Notes

• —This is an abliterated (uncensored) model. The base model has had safety training removed. Use responsibly.
• —Vision tower is kept in BF16 — multimodal capabilities are preserved at full precision.
• —NVFP4 is a Blackwell-specific format. This checkpoint will not work on Ampere/Hopper GPUs.

Credits

• —Base model: huihui-ai (abliteration)
• —Original model: Google DeepMind (Gemma 4)
• —NVFP4 quantization & benchmarking: Lna-Lab
• —Blackwell NVFP4 GEMM kernels: lna-lab/blackwell-geforce-nvfp4-gemm
• —Quantization framework: vllm-project/llm-compressor
• —Quantization method follows RedHatAI/gemma-4-26B-A4B-it-NVFP4 (proven path)

Support the Base Model Author

If you find this model useful, please consider supporting huihui-ai — the creator of the abliterated base model:

• —Ko-fi: https://ko-fi.com/huihuiai
• —Bitcoin: bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge