happypatrick/Qwen3.5-122B-A10B-heretic-int4-AutoRound

Qwen3.5-122B-A10B-heretic-int4-AutoRound

The first INT4 AutoRound quantization of the Heretic (uncensored) Qwen3.5-122B — with vision preserved.

A 4-bit symmetric quantization of trohrbaugh/Qwen3.5-122B-A10B-heretic, generated using Intel AutoRound v0.12.0. The original multimodal architecture (Qwen3_5MoeForConditionalGeneration) is fully preserved — text, vision, video, and reasoning all work.

Model Lineage

This model has a three-step provenance chain:

Qwen/Qwen3.5-122B-A10B          ← Qwen team's flagship 122B MoE (BF16, 234GB)
  └─► trohrbaugh/heretic         ← Abliterated with Heretic v1.2.0 (KL=0.0916, 9/100 refusals)
       └─► THIS MODEL            ← INT4 AutoRound quantization (63GB, vision preserved)

1. 1.[Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B) — Qwen team's 122B Mixture-of-Experts model with DeltaNet hybrid attention, 256 experts (8 active + 1 shared per token), native 262K context, and multimodal vision support.

1. 1.[trohrbaugh/Qwen3.5-122B-A10B-heretic](https://huggingface.co/trohrbaugh/Qwen3.5-122B-A10B-heretic) — Abliterated (uncensored) variant using Heretic v1.2.0 by p-e-w. Uses parametrized directional ablation with interpolated direction index (41.20), applied to attn.o_proj and mlp.down_proj. Reduces refusals from 99/100 to 9/100 while maintaining low KL divergence (0.0916).

1. 1.This model — INT4 AutoRound quantization targeting unified-memory GPU systems (NVIDIA DGX Spark / ASUS Ascent GX10). All 333 vision weights, 192 shared expert layers, 48 MoE gate layers, and lm_head preserved at full precision. Text module quantized to INT4 W4G128.

Quantization Details

What Was Preserved (NOT Quantized)

Keeping critical layers at full precision is essential for MoE model quality:

• —333 vision encoder weights (model.visual.blocks.*) — BF16
• —192 shared expert projections (48 layers × gate_proj, up_proj, down_proj) — FP16
• —48 shared expert gates (shared_expert_gate) — FP16
• —48 MoE routing gates — FP16
• —lm_head — original precision

The shared expert is activated for every token (unlike routed experts), so preserving its precision is critical for maintaining output quality. The vision encoder is kept at BF16 to preserve multimodal capabilities.

Why AutoRound?

AutoRound uses sign-gradient descent to optimize weight rounding decisions, achieving better accuracy than RTN (round-to-nearest) and competitive results with GPTQ/AWQ while being faster and more robust. The GPTQ-compatible packing format (auto_round:auto_gptq) means this model works with vLLM's optimized Marlin CUDA kernels out of the box.

Model Size Comparison

The 9 GB difference from Intel's canonical INT4 is because the Heretic fork does not include MTP (Multi-Token Prediction) weights that are present in the base Qwen model.

Performance (Measured on DGX Spark Cluster)

Tested on a 2-node NVIDIA DGX Spark cluster (2× Grace Blackwell GB10, 128GB unified memory each) with tensor parallelism over 200Gbps RoCE RDMA:

Head-to-Head: Heretic INT4 vs Cardinal (Canonical) INT4

We ran identical test batteries against both this model and Intel/Qwen3.5-122B-A10B-int4-AutoRound on the same hardware:

Verdict: Quality is essentially identical. The Heretic is slightly faster on reasoning tasks (27.5 vs 24.9 tok/s). Both models produced complete, working code with proper error handling and TypeScript types. The Heretic's code generation included a clock.ts time abstraction — a marginally more mature architectural pattern.

Quick Start

vLLM (Recommended)

pip install vllm>=0.17.0

vllm serve happypatrick/Qwen3.5-122B-A10B-heretic-int4-AutoRound \
  --served-model-name qwen3.5-122b \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.85 \
  --enforce-eager \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --trust-remote-code

Then query with any OpenAI-compatible client:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5-122b",
    "messages": [{"role": "user", "content": "Write a Python quicksort implementation"}],
    "temperature": 0.6
  }'

HuggingFace Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "happypatrick/Qwen3.5-122B-A10B-heretic-int4-AutoRound"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    trust_remote_code=True,
)

messages = [{"role": "user", "content": "Hello!"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

DGX Spark / ASUS Ascent GX10 Deployment Guide

This model was specifically tested on NVIDIA DGX Spark (also sold as ASUS Ascent GX10) — a desktop workstation with a Grace Blackwell GB10 SoC and 128 GB unified CPU+GPU memory. Here's how to run it:

Single Node (128 GB Unified Memory)

# Conservative settings — leaves room for ~50GB KV cache
vllm serve /path/to/heretic-int4-autoround-v2 \
  --served-model-name qwen3.5-122b \
  --host 0.0.0.0 --port 8000 \
  --gpu-memory-utilization 0.75 \
  --max-model-len 131072 \
  --enforce-eager \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --trust-remote-code

Warning: Do NOT set --gpu-memory-utilization above 0.85 on unified memory systems. Higher values can cause system freezes. Start at 0.75 and increase carefully.

Dual Node (2× DGX Spark, 256 GB Total)

For two DGX Sparks connected via 200Gbps RoCE RDMA (the setup we tested on):

# Node 2 (worker) — start first
docker run -d --name vllm-worker \
  --gpus all --network=host --ipc=host --privileged \
  --shm-size 10.24g \
  -v /path/to/models:/models \
  -e NCCL_SOCKET_IFNAME=enP2p1s0f0np0 \
  -e NCCL_IB_HCA=roceP2p1s0f0 \
  -e NCCL_P2P_DISABLE=1 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3600 \
  vllm/vllm-openai:latest \
  bash -c "ray start --address=<NODE1_IP>:6399 --num-gpus=1 --node-ip-address=<NODE2_IP> --block"

# Node 1 (head) — start second
docker run -d --name vllm-head \
  --gpus all --network=host --ipc=host --privileged \
  --shm-size 10.24g \
  -v /path/to/models:/models \
  -e NCCL_SOCKET_IFNAME=enP2p1s0f0np0 \
  -e NCCL_IB_HCA=roceP2p1s0f0 \
  -e NCCL_P2P_DISABLE=1 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3600 \
  vllm/vllm-openai:latest \
  bash -c "ray start --head --port=6399 --num-gpus=1 --node-ip-address=<NODE1_IP> && \
  vllm serve /models/heretic-int4-autoround-v2 \
    --served-model-name qwen3.5-122b \
    --host 0.0.0.0 --port 8000 \
    --tensor-parallel-size 2 \
    --distributed-executor-backend ray \
    --max-model-len 225000 \
    --gpu-memory-utilization 0.50 \
    --enforce-eager \
    --enable-prefix-caching \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml \
    --trust-remote-code"

DGX Spark Tips & Gotchas

• —`--shm-size 10.24g` is critical for multi-node Ray — without it, model loading can hang at ~57% (NCCL deadlock)
• —`VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3600` prevents non-streaming requests from timing out at the default 300s for TP>1 configurations
• —`--enforce-eager` is recommended — CUDA graph compilation is slow on GB10 and the memory savings aren't worth it at these model sizes
• —`NCCL_P2P_DISABLE=1` is required for GB10's unified memory architecture — P2P transfers hang
• —Set `vm.swappiness=1` at the OS level (sysctl vm.swappiness=1) to prevent the kernel from swapping GPU-mapped pages
• —The 200Gbps DAC cable (RoCE RDMA) between nodes provides enough bandwidth for TP=2 tensor parallelism with minimal overhead

Reproduce the Quantization

Requirements

• —GPU: 2× NVIDIA H200 SXM (or similar — needs ~80GB combined VRAM + 450GB RAM)
• —Time: ~4 hours 12 minutes
• —Cost: ~$33.50 on RunPod ($7.18/hr for 2× H200 SXM pod)

Setup

# Create a RunPod instance with 2x H200 SXM, 2TB RAM
# SSH in and install dependencies

pip install torch>=2.5.0
pip install git+https://github.com/intel/auto-round.git    # v0.12.0+ (PyPI is too old)
pip install git+https://github.com/huggingface/transformers.git  # v5.3.0+

# Download the Heretic base model (~234GB, takes a while)
huggingface-cli download trohrbaugh/Qwen3.5-122B-A10B-heretic \
  --local-dir /workspace/heretic-bf16

Quantize

auto-round \
  --model_name "/workspace/heretic-bf16" \
  --output_dir "/workspace/output/heretic-int4" \
  --ignore_layers shared_expert \
  --device_map "auto"

That's it. AutoRound handles everything else — calibration data, iteration count, group size, and format are all set to sensible defaults.

What Happens During Quantization

• —AutoRound processes each transformer layer sequentially (~5.3 minutes per layer, 48 layers)
• —For each layer, it uses sign-gradient descent to optimize rounding decisions (INT4 W4G128)
• —Shared expert layers are automatically skipped (kept at FP16) due to --ignore_layers shared_expert
• —Vision encoder weights are preserved at BF16 (they live outside model.language_model.layers)
• —Peak memory: 453.8 GB RAM, 35-45 GB VRAM per GPU
• —Each layer shows 50-60% loss reduction from the optimization

Learnings from the Process

1. 1.Use `auto-round` from git, not PyPI. The PyPI release (0.10.2) doesn't support Qwen3.5 MoE models. You need the merged PR #1476 which adds Qwen3_5MoeForConditionalGeneration support.

1. 1.Transformers must also be from git. The released version doesn't have the Qwen3.5 model class yet.

1. 1.PyTorch 2.4 will fail with AttributeError: 'Linear' object has no attribute 'set_submodule'. Use PyTorch 2.5+.

1. 1.`--ignore_layers shared_expert` is important. The shared expert is activated for every single token (unlike the 255 routed experts where only 8 fire per token). Quantizing it hurts output quality disproportionately. Intel uses the same flag for the canonical model.

1. 1.`--device_map "auto"` distributes across GPUs correctly. With 2× H200, the model splits cleanly and quantization runs in parallel where possible.

1. 1.The Heretic fork has no MTP weights. This is why our output is 63GB instead of the 72GB you'd get from quantizing the canonical Qwen model. Not a quality issue — MTP (Multi-Token Prediction) is a speculative decoding feature, not a capability.

1. 1.Vision is automatically preserved. Because we set block_name_to_quantize to model.language_model.layers, the vision encoder (model.visual.*) is never touched. You get full multimodal support in the quantized model.

Abliteration Notice

This model is based on the "Heretic" variant of Qwen3.5-122B-A10B, which has had safety refusals significantly reduced using directional ablation (Heretic v1.2.0 by p-e-w).

• —The base model refused 99/100 test prompts. The Heretic variant refuses 9/100.
• —KL divergence from the original: 0.0916 (low — general capabilities well preserved).
• —Abliteration method: parametrized directional ablation with interpolated direction index, applied to attn.o_proj and mlp.down_proj across all 48 layers.

This model will follow instructions that the original Qwen model would refuse. Please use responsibly. The creators are not responsible for misuse.

Ethical Considerations and Limitations

• —This is an uncensored model. It has reduced safety guardrails compared to the original Qwen3.5-122B-A10B.
• —Quantization to INT4 introduces a small quality degradation compared to BF16, though our testing shows negligible impact on code generation, reasoning, and tool calling tasks.
• —The model may generate biased, incorrect, or harmful content. Users should implement appropriate safety measures for their applications.
• —This model should not be used to generate content that could cause harm to individuals or groups.

License

This model inherits the Apache 2.0 license from Qwen/Qwen3.5-122B-A10B.

Acknowledgments

• —[Qwen Team](https://huggingface.co/Qwen) for the incredible Qwen3.5-122B-A10B base model
• —[trohrbaugh](https://huggingface.co/trohrbaugh) for the Heretic abliteration
• —[p-e-w](https://github.com/p-e-w) for the Heretic abliteration tool
• —[Intel](https://github.com/intel/auto-round) for AutoRound and for demonstrating the --ignore_layers shared_expert recipe on the canonical model
• —[RunPod](https://runpod.io) for affordable H200 GPU access
• —NVIDIA for the DGX Spark platform

Citation

If you use this model, please cite both the AutoRound quantization method and the original Qwen model:

@article{cheng2024optimize,
  title={Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs},
  author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},
  journal={arXiv preprint arXiv:2309.05516},
  year={2024}
}

@misc{qwen3.5,
  title={Qwen3.5 Technical Report},
  author={Qwen Team},
  year={2025},
  url={https://qwenlm.github.io/blog/qwen3.5/}
}

Quantized on 2026-03-15 using RunPod 2× H200 SXM. Tested on a 2-node DGX Spark cluster with 200Gbps RoCE RDMA.