hipfire-models/qwen3.5-4b

Qwen3.5-4B for hipfire

Pre-quantized Qwen3.5-4B (DeltaNet hybrid) for hipfire, a Rust-native LLM inference engine for AMD RDNA GPUs.

Quantized from Qwen/Qwen3.5-4B.

Files

Speeds are forward-only tok/s on the listed AMD GPU. ⭐ MQ4 ships with a mandatory byte-exact greedy quality gate (9 reference token streams).

Usage

# Install hipfire
curl -L https://raw.githubusercontent.com/Kaden-Schutt/hipfire/master/scripts/install.sh | bash

# Pull and run any variant
hipfire pull qwen3.5:4b            # HF4 (default — fastest)
hipfire pull qwen3.5:4b-mq4        # MQ4 (quality-gated, near-Q8 output)
hipfire pull qwen3.5:4b-hf6        # HF6 (highest quality, ~15% slower)

hipfire run qwen3.5:4b-mq4 "Hello"

Quantization Formats

• —HF4 (HFQ4-G256) — flat 4-bit, 256-weight groups (~0.53 B/w including per-group scale + zero). Best raw tok/s. Same storage layout as Q4KM in llama.cpp but without the K-quant block descriptors.

• —HF6 (HFQ6-G256) — flat 6-bit, 256-weight groups (~0.78 B/w). Highest quality, ~15% slower than HF4. Use this if you have VRAM headroom and want the smallest accuracy loss vs FP16.

• —MQ4 (MagnumQuant 4-bit) ⭐ — FWHT-rotated 4-bit. Storage layout identical to HF4 (4.25 B/w), but the weights are pre-rotated through a Walsh–Hadamard transform at quantization time, and the input x vector is rotated through the same transform on the fly during the GEMV. The rotation flattens outliers, dramatically improving the quantization-error distribution. Result: roughly Q8-grade output quality at Q4 bandwidth.

Every commit that touches kernel or forward-pass code in the hipfire repo is gated against MQ4 byte-exact greedy decoding of 9 reference (model, prompt) pairs — see tests/quality-baselines and scripts/quality-gate.sh. Any silent numerical regression in the forward pass is caught at commit time.

All formats embed the tokenizer and model config inside the model file — no separate tokenizer.json download needed.

About hipfire

Rust + HIP inference engine for AMD consumer GPUs (RDNA1–RDNA4). No Python in the hot path. The 0.1.4-alpha branch lands a kernel-fusion overhaul that roughly doubles forward speed on Qwen3.5 across the lineup vs the previous release.

• —GitHub: Kaden-Schutt/hipfire
• —All models: docs/MODELS.md

License

Model weights subject to the original Qwen license. hipfire engine: MIT.