YuYu1015/Huihui-Qwen3.5-27B-abliterated-int4-AutoRound

Huihui-Qwen3.5-27B-abliterated-int4-AutoRound

English | 繁體中文

English

INT4 AutoRound quantization of huihui-ai/Huihui-Qwen3.5-27B-abliterated, optimized for NVIDIA DGX Spark (GB10 SM121) with Marlin INT4 kernel acceleration.

Model Details

Quantization Details

Layers Preserved in BF16

The following layers are not quantized to preserve model quality:

Speculative Decoding

DFlash (requires separate drafter model):

--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.5-27B-DFlash", "num_speculative_tokens": 16}'

Note: The DFlash drafter was trained on the original Qwen3.5-27B. Acceptance rate on the abliterated variant may be lower than on the original model.

Serving with vLLM

vllm serve /path/to/model \
    --quantization gptq_marlin \
    --served-model-name qwen3.5-27b \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --kv-cache-dtype auto \
    --gpu-memory-utilization 0.90 \
    --max-model-len 65536 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --trust-remote-code \
    --language-model-only

DGX Spark (SM121) Compatibility Notes

• —Use --quantization gptq_marlin for Marlin INT4 kernel (Dense model, not MoE)
• —FP8 KV cache is not compatible with GDN non-causal attention layers; use --kv-cache-dtype auto
• —NVFP4 is not supported on SM121 (missing cvt.e2m1x2 instruction)
• —Runtime FP8 (--quantization fp8) is not compatible with DFlash
• —--language-model-only skips vision encoder profiling for text-only inference
• —Clear page cache before starting on UMA: sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

Safety Warning

This model has safety filtering removed (abliterated) and may generate inappropriate content. Users are solely responsible for all consequences arising from its use.

Credits

• —Original Model: Qwen/Qwen3.5-27B by Alibaba Qwen Team
• —Abliteration: huihui-ai
• —INT4 Quantization: YuYu1015 on NVIDIA DGX Spark (GB10)
• —Quantization Tool: Intel AutoRound

繁體中文

huihui-ai/Huihui-Qwen3.5-27B-abliterated 的 INT4 AutoRound 量化版本，針對 NVIDIA DGX Spark (GB10 SM121) 最佳化。

模型資訊

量化詳情

保留 BF16 的層

以下層未被量化以保持模型品質：

DGX Spark (SM121) 相容性說明

• —使用 --quantization gptq_marlin 啟用 Marlin INT4 kernel（Dense 模型，非 MoE）
• —FP8 KV cache 與 GDN non-causal attention 不相容，請使用 --kv-cache-dtype auto
• —NVFP4 在 SM121 上不支援（缺少 cvt.e2m1x2 指令）
• —Runtime FP8（--quantization fp8）與 DFlash 不相容
• —UMA 架構啟動前請先清除 page cache：sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

安全警告

此模型已移除安全過濾機制（abliterated），可能產生不當內容。使用者須自行承擔所有風險與法律責任。

致謝

• —原始模型：Qwen/Qwen3.5-27B，Alibaba Qwen 團隊
• —去審查：huihui-ai
• —INT4 量化：YuYu1015，於 NVIDIA DGX Spark (GB10) 上完成
• —量化工具：Intel AutoRound