abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw

Qwen3.5-9B-Abliterated-GDN-Hybrid — EXL3 6.0bpw

EXL3 (exllamav3) quantization of huihui-ai/Huihui-Qwen3.5-9B-abliterated at 6.0 bits per weight.

Quantized by ABC&D System Inc for production real-time inference on a single RTX 3090.

Model Details

Architecture — Why This Model Is Different

Qwen3.5 is not a standard transformer. It uses a hybrid attention design that alternates between two fundamentally different attention mechanisms:

GatedDeltaNet Linear Attention (layers 0, 1, 2, 4, 5, 6, 8, 9, 10, ...)

• —O(n) complexity instead of O(n²) — scales linearly with sequence length
• —Recurrent state via conv1d kernel (kernel dim 4)
• —16 key heads × 128 dim, 32 value heads × 128 dim
• —Gated with learned input/output projections + beta decay

Full Quadratic Attention (layers 3, 7, 11, 15, 19, 23, 27, 31)

• —Standard grouped-query attention (16 heads, 4 KV heads, head dim 256)
• —RoPE with partial rotary factor 0.25, theta 10M
• —Provides global context every 4th layer

Layer Pattern

[linear, linear, linear, FULL, linear, linear, linear, FULL, ...] × 4 = 32 layers

24 GatedDeltaNet layers + 8 full attention layers. The linear layers handle local/sequential patterns efficiently while the full attention layers provide global context anchoring.

Multi-Token Prediction (MTP)

The model includes a 1-layer MTP head for predicting multiple next tokens simultaneously, enabling faster inference throughput than standard single-token autoregressive decoding.

Text-Only Weights

Despite the architecture string Qwen3_5ForConditionalGeneration (which Alibaba uses for all Qwen3.5 variants including VL), this model contains zero vision weights. No ViT encoder, no image projector. The architecture class name reflects the MTP head and hybrid design, not multimodal capability.

Quantization Details

Quantized with exllamav3 v1.1.0.

• —Format: EXL3 with trellis-based weight encoding
• —Bits per weight: 6.0 (all linear layers uniform)
• —Head bits: 6
• —Codebook: mul1
• —Why EXL3 over GGUF: Native tensor core utilization on Ampere GPUs (RTX 3090/4090). GGUF dequantizes to FP16 at runtime, wasting tensor cores. EXL3 operates natively, yielding ~1.5x faster inference at equivalent quality.

Recommended Serving

TabbyAPI (recommended)

# config.yml
network:
  host: 0.0.0.0
  port: 11435

model:
  model_dir: /path/to/models
  model_name: Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw
  max_seq_len: 4096
  gpu_split: [24]
  cache_mode: FP16

CUDA_VISIBLE_DEVICES=0 python3 main.py --config config.yml

Important: Disable Thinking Mode

Qwen3.5 has a built-in chain-of-thought reasoning mode that is on by default. For real-time applications (voice, chat, API), you must disable it or the model will consume your entire token budget on internal reasoning before producing visible output.

Add to your system prompt:

/no_think

Or set enable_thinking: false in your chat template parameters.

Note on exllamav3 Raw API

The raw exllamav3 Generator Python API may fail to initialize GatedDeltaNet recurrent state (conv_state on meta device error). TabbyAPI handles this initialization correctly. Use TabbyAPI as the serving layer rather than calling exllamav3 directly.

Hardware Requirements

Performance (tested on RTX 3090)

• —Response latency: 300–500ms per completion
• —Suitable for real-time voice pipelines with STT overhead
• —First request ~14ms (cache warmup, empty response)

Use Cases

This quantization was built for low-latency, single-user dedicated inference — specifically real-time voice AI over SIP/phone calls. The abliteration ensures zero self-censorship in commercial conversation contexts.

Good for:

• —Real-time voice assistants and phone agents
• —Single-GPU dedicated inference servers
• —Applications requiring uncensored, natural conversational output
• —Latency-sensitive pipelines (sub-600ms budget)

Not ideal for:

• —Multi-user batched serving (use AWQ + vLLM instead)
• —Vision/multimodal tasks (no vision weights)
• —Applications requiring safety guardrails (abliterated model)

Credits

• —Base model: Qwen/Qwen3.5-9B by Alibaba Cloud (Apache 2.0)
• —Abliteration: huihui-ai via remove-refusals-with-transformers
• —EXL3 quantization: ABC&D System Inc using exllamav3 by turboderp
• —Serving: TabbyAPI by theroyallab

Usage Warnings

This model has had its safety filtering surgically removed (abliterated). It will comply with any instruction without refusal. Users are solely responsible for ensuring their use complies with applicable laws and ethical standards. See huihui-ai's original warnings for full details.