jc-builds/Qwen3.5-9B-Q4_K_M-GGUF

Qwen3.5-9B Q4KM GGUF

4-bit quantized GGUF of Qwen/Qwen3.5-9B optimized for on-device iOS inference via llama.cpp. The most capable model you can run on an iPhone.

Running It

This model is available directly in the HaploAI iOS app (v1.18+). Download it from the model selection page.

Key Features

• —Best-in-Class On-Device AI: Matches or beats models 9-13x its size
• —Thinking Mode: <think>...</think> chain-of-thought reasoning
• —Hybrid Architecture: Gated DeltaNet + Attention for efficient, high-quality inference
• —Natively Multimodal: Trained with early vision fusion
• —Massive Context: 262K native, extendable to 1M+ with YaRN

Benchmarks

Qwen3.5-9B vs Models 9-13x Larger

[image]

The 9B model beats Qwen3-80B on GPQA Diamond (81.7 vs 77.2), IFEval (91.5 vs 88.9), and HMMT math (83.2 vs 73.7). It also outperforms GPT-OSS-120B on MMLU-Pro (82.5 vs 80.8) and GPQA Diamond (81.7 vs 80.1).

MMLU-Pro: Size Class Comparison

On-Device Inference Speed

[image]

Vision Capabilities

[image]

The 9B model outperforms the dedicated Qwen3-VL-30B (3x its size) on MMMU, MMMU-Pro, MathVision, OmniDocBench, and VideoMME.

Full Benchmark Table

Vision Benchmarks

Device Compatibility

Note: The 9B model at 5.3 GB requires devices with 8 GB+ RAM. For older devices, use the Qwen3.5-4B instead.

Usage

With llama.cpp

# Download
huggingface-cli download jc-builds/Qwen3.5-9B-Q4_K_M-GGUF Qwen3.5-9B-Q4_K_M.gguf

# Run (with thinking mode)
./llama-cli -m Qwen3.5-9B-Q4_K_M.gguf -p "Prove that there are infinitely many primes." -ngl 99

With Ollama

ollama run qwen3.5:9b

Prompt Format

Uses ChatML format:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Hello!<|im_end|>
<|im_start|>assistant

Thinking Mode

<|im_start|>assistant
<think>
Let me reason through this carefully...
First, assume there are finitely many primes p1, p2, ..., pn.
Consider N = p1 * p2 * ... * pn + 1.
N is not divisible by any pi, so either N is prime or has a prime factor not in our list.
This contradicts our assumption.
</think>
There are infinitely many primes. Here is Euclid's classic proof...

Architecture Details

Qwen3.5 introduces a hybrid Gated DeltaNet + Gated Attention architecture:

• —3:1 ratio: 3 layers of Gated DeltaNet (linear attention) per 1 layer of full softmax attention
• —32 total layers: 8 blocks x (3 DeltaNet + 1 Attention)
• —Hidden dimension: 4,096
• —Near-constant memory: DeltaNet layers maintain bounded memory
• —GQA: 16 query heads, 4 KV heads for attention layers
• —FFN intermediate: 12,288
• —RoPE: theta=10,000,000 with YaRN extension

Why Qwen3.5-9B?

This model represents a paradigm shift in on-device AI:

1. 1.9B params that beat 80B: On GPQA Diamond, IFEval, and math benchmarks
2. 2.Hybrid attention is the future: DeltaNet layers provide near-constant memory, enabling huge context on mobile
3. 3.Natively multimodal: No separate vision encoder needed for basic image understanding
4. 4.201 languages: Broadest language support in its class
5. 5.Apache 2.0: Fully open, commercially usable

Credits

• —Original model: Qwen/Qwen3.5-9B by Alibaba Cloud
• —GGUF conversion: Unsloth
• —Quantization: Q4KM via llama.cpp
• —Optimized for iOS: jc-builds