Modelpublic

nerkyor/Qwen3.5-4B-GGUF-imatrix

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

0likes49downloads

Model Card

Qwen3.5-4B Q4KM (imatrix) - Lynn calibration

This repository ships an imatrix-calibrated Q4_K_M GGUF of the official Qwen/Qwen3.5-4B BF16 weights, built on DGX Spark (GB10, sm_121) by the Lynn team.

This is a pure quantization of the upstream model. It is not a distillation and not a Lynn-native NVFP4/W4A8 checkpoint.

Files

File	Size	SHA256	Role
`Qwen3.5-4B-Q4_K_M-imatrix.gguf`	2.6 GB	`7abaf02bbe25c608deb308db526766f761ad4fb85c512a69ff36520c4b304b23`	llama.cpp / Ollama / LM Studio GGUF weights
`Qwen3.5-4B.imatrix`	3.5 MB	`863a93c58a14925b58303a369d9bb155411b40d52fde121f195eaf7a6691f07c`	imatrix calibration data used for quantization

Build Details

—Source weights: Qwen/Qwen3.5-4B official BF16
—Converter: llama.cpp convert_hf_to_gguf.py --outtype f16
—Calibration: Lynn Chinese + English + code mix, 256 chunks, 512 ctx
—Quantizer: llama-quantize --imatrix Qwen3.5-4B.imatrix ... Q4_K_M
—Built on: DGX Spark (GB10, sm_121), 2026-05-24

Evaluation

V8/V9, MMLU500, and GPQA Diamond thinking-on evaluations are running on Spark. Those artifacts and scores will be added in a follow-up update.

Run

bash

llama-server \
  -m Qwen3.5-4B-Q4_K_M-imatrix.gguf \
  --host 0.0.0.0 --port 18099 \
  --ctx-size 32768 --n-gpu-layers 999 \
  --jinja --reasoning on

The GGUF embeds the upstream Qwen3.5 chat template, including thinking-mode support via chat_template_kwargs.enable_thinking.

License

Apache 2.0, inherited from Qwen/Qwen3.5-4B.