Modelpublic

joshebbs/qwen3.6-35b-abliterated-nvfp4-modelopt

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

1likes384downloads

Model Card

Qwen3.6-35B-A3B Abliterated NVFP4 (ModelOpt)

Abliterated and NVFP4-quantized version of Qwen/Qwen3.6-35B-A3B. Optimized for NVIDIA Blackwell (DGX Spark / GB10).

Performance

2x NVIDIA DGX Spark (GB10, SM121 Blackwell) — TP=2, CUDA Graphs

Metric	NVFP4 TP=2	FP8 TP=2	Improvement
Decode tg128 c1	83.21 tok/s	77.31 tok/s	+8%
Peak tg128 c1	84.00 tok/s	—	—
Decode tg128 c4 total	194.54 tok/s	90.58 tok/s	+115%
Peak tg128 c4	249.33 tok/s	—	—
Prefill pp2048 c1	8,033 tok/s	5,636 tok/s	+43%
TTFT pp2048 c1	234 ms	320 ms	-27%
Model size	20 GB	35 GB	-43%
KV cache available	81.6 GB	74.5 GB	+10%

Single DGX Spark — CUDA Graphs

Metric	Value
Decode tg128 c1	63.21 tok/s
Decode tg128 c4 total	143.00 tok/s
Peak tg128 c4	178.67 tok/s
Prefill pp2048	7,083 tok/s

What's in this model?

1.Abliteration: Refusal directions removed from layers 14-26 using norm-preserving biprojected abliteration
2.NVFP4 Quantization: Quantized with NVIDIA ModelOpt (v0.43.0) using 256 calibration samples from Open-Platypus, batch_size=64
3.Format: ModelOpt FP4 — compatible with vLLM (--quantization modelopt_fp4)

Model Details

—Base Model: Qwen/Qwen3.6-35B-A3B (Apache 2.0)
—Architecture: MoE, 35B total params, 3B active per token, 256 experts (8 routed + 1 shared)
—Quantization: NVFP4 on language model linear layers. Vision encoder, gates, conv1d, lm_head excluded.
—Size: ~20GB (vs 67GB BF16, vs 35GB FP8)
—Context: 32K (configurable)

Usage with vLLM

Single Node

bash

vllm serve joshebbs/qwen3.6-35b-abliterated-nvfp4-modelopt \
    --quantization modelopt_fp4 \
    --trust-remote-code \
    --dtype auto \
    --kv-cache-dtype fp8 \
    --attention-backend flashinfer \
    -tp 1

Multi-Node (2x DGX Spark)

bash

vllm serve joshebbs/qwen3.6-35b-abliterated-nvfp4-modelopt \
    --quantization modelopt_fp4 \
    --trust-remote-code \
    --dtype auto \
    --kv-cache-dtype fp8 \
    --attention-backend flashinfer \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    -tp 2 --nnodes 2 --node-rank 0 \
    --master-addr <HEAD_IP> --master-port 29501

Important Notes

—Vision: This checkpoint includes the ConditionalGeneration config for compatibility, but vision weights are not included. The visual encoder dimensions (4304) are not NVFP4-aligned for TP>1. For vision support, use the base FP8 model.
—First startup: FlashInfer JIT compilation takes ~25-30 minutes on first run. Subsequent starts use cached kernels (~5-7 minutes).
—Multi-node timeout: The mp distributed backend requires patching store_timeout in vllm/distributed/utils.py from 300s to 1800s for first-run JIT compilation across nodes.
—FLASHINFER_NVCC_THREADS: Set to 16 for faster JIT compilation on multi-core systems.

Quantization Recipe

Quantized on NVIDIA H200 via Vast.ai in 7.4 minutes ($3.31):

python

import modelopt.torch.quantization as mtq
quant_config = mtq.NVFP4_DEFAULT_CFG
model = mtq.quantize(model, quant_config, forward_loop=calibrate_loop)
# calibrate_loop: 256 samples, batch_size=64, max_seq_length=1024

Abliteration Details

—Tool: jim-plus/llm-abliteration
—Measurement: Projected mode with flash attention on BF16 model
—Layers: 14-26 (strongest refusal signal, signal quality 0.14-0.22)
—Scale: 1.0, Sparsity: 0.0

Hardware Tested

—2x NVIDIA DGX Spark (GB10 Blackwell, 128GB unified memory, 200Gb/s QSFP RoCE)
—NVIDIA H200 (144GB HBM3e) — validation only