piscesbody/Qwen3.8-27B-Uncensored-NVFP4

Qwen3.8-27B-Uncensored-NVFP4 (mixed-precision, vision retained)

English | 中文

FP4-quantized build of orcarouter/Qwen3.8-27B-Uncensored (BF16), preserving the abliteration edits byte-for-byte — this is a quantization of the uncensored weights, NOT of the official Qwen3.8. The vision tower (BF16) and the full multimodal config are included, so the model works for both text-only and image/video serving.

Quantization recipe (mixed precision, mirrors nvidia/Qwen3.8-27B-NVFP4)

• —Producer: NVIDIA Model Optimizer (modelopt) 0.46.0
• —Calibration: cnn_dailymail (3.0.0), 512 samples × 2048 tokens, max algorithm
• —KV cache: bf16/fp8 both work (no kv scales shipped — serve with default kv dtype)

Eval (vs the FP8 source build, on our own stack)

• —L1 reasoning suite (semantic+arithmetic): 6/6 (source: 6/6)
• —L2 contamination-resistant suite (AIME 2025 pair-counting, combinatorics, number theory, LIS coding, Chinese logic, strict format): 6/6 (source: 6/6)
• —Refusal behaviour: the quantization source is the abliterated BF16 checkpoint, so the abliteration edits are preserved by construction. No refusal benchmark (e.g. AdvBench/StrongREJECT) was run by the uploader on this quantized build.

Serving

SGLang (validated, our fork of main ~2026-08-31 with modelopt MIXED support)

python -m sglang.launch_server \
  --model-path /path/to/Qwen3.8-27B-Uncensored-NVFP4-v2 \
  --language-only `# optional: text-only serving skips vision weights` \
  --tp 1

• —On Ada (SM89, e.g. RTX 4090): FP4 GEMMs run via Marlin W4A16 automatically. Set SGLANG_DISABLE_SILU_FP4_QUANT_FUSION=1 (the fused SiLU+FP4 kernel has no Ada backend in current flashinfer and will crash startup with Invalid backend: 89 otherwise).
• —On Blackwell: native FP4 tensor cores via flashinfer/cutlass backends.

The modelopt_fp4 (W4A4) compressed-tensors builds published elsewhere may refuse to load on Ada (min_capability=100). This checkpoint ships the W4A16-serialized form, which runs on SM89.

vLLM

MIXED_PRECISION modelopt checkpoints should load like nvidia/Qwen3.8-27B-NVFP4 (vllm ≥ 0.27). Not validated by us — feedback welcome.

Hardware notes (RTX 4090 48G, sglang)

Decode wins come from FP4 weight bandwidth + FP8 attention projections; prefill pays a Marlin dequant tax on Ada only (Blackwell has native FP4 compute). Measured on one RTX 4090 48G, TP1, DFlash2 speculative decoding K=8, fp8 KV cache.

Files

• —model-0000{1,2}-of-00003.safetensors — quantized language model (20.2 GB)
• —model-00003-of-00003.safetensors — BF16 vision tower, 333 tensors (0.92 GB, from the base checkpoint)
• —hf_quant_config.json — modelopt MIXED_PRECISION manifest (per-layer recipe)
• —LICENSE — Apache-2.0, inherited from the base checkpoint
• —README.zh.md — Chinese version of this document
• —full tokenizer + processor + chat template files included

Reproduction scripts (see next section):

• —quantization_reproduce.py — step 1: quantize + export
• —vision_merge.py — step 2: copy the BF16 vision tower in
• —make_composite_config.py — step 3: rebuild config.json into the composite VL shape

Reproducing this build

# 1. quantize (mixed precision) -> writes a flat-config export
python quantization_reproduce.py \
    --src /path/to/Qwen3.8-27B-Uncensored \
    --dst /path/to/out

# 2. merge the BF16 vision tower (no requant; file-level tensor copy)
python vision_merge.py --base /path/to/Qwen3.8-27B-Uncensored --dst /path/to/out

# 3. rebuild config.json into the composite VL shape
python make_composite_config.py --base /path/to/Qwen3.8-27B-Uncensored --dst /path/to/out

Step 3 is not optional for SGLang multimodal serving. export_hf_checkpoint writes a flat text config (architectures: ["Qwen3_5ForCausalLM"]), but the encoder/decoder split whitelist used by --language-only only accepts the composite name Qwen3_5ForConditionalGeneration with a nested text_config. A flat export loads as text-only but is rejected by the split hook.

Two modelopt 0.46.0 gotchas the scripts already handle (both cost a failed run each if you hit them from scratch):

• —`{"algorithm": "max"}` is required in mtq.quantize(...). Without it, calibrate() dispatches to NoneCalibrateMode and silently skips the forward loop — the run "succeeds" in seconds with every amax unset.
• —`_amax` export bug. The FP8 exporter reads weight_quantizer._amax, but TensorQuantizer now exposes it as the amax property. quantization_reproduce.py wraps _export_quantized_weight to rebind it temporarily.

Requirements: torch, transformers, modelopt==0.46.0, one CUDA GPU. The BF16 weights are ~54 GB and do not fit a 48 GB card, so device_map spills to host RAM (--gpu-mem / --cpu-mem control the split; defaults 24GiB / 80GiB).

License & attribution

Apache-2.0, same as the base. Abliteration by orcarouter (see base repo); quantization pipeline by the uploader. If you redistribute, keep the base model attribution.