webhie/Qwen3.8-27B-Q4-AutoRound-Code-GGUF

Qwen3.8-27B Q4_0 AutoRound Code GGUF

A mixed GGUF Q4_0 quantization of `Qwen/Qwen3.8-27B`, produced with Intel's AutoRound. The main linear weights use GGUF Q4_0; output.weight is retained as Q6_K, while normalization and selected state-space auxiliary tensors remain F32.

Key difference from a Transformers-format AutoRound checkpoint: This release is packaged for direct use with [llama.cpp](https://github.com/ggml-org/llama.cpp). It includes the language-model GGUF and the separate mmproj-model.gguf vision projector. The quantization used the auto-round-best recipe with 512 calibration samples from a code-heavy, mixed code/math/science calibration set. No benchmark results are reported for this release.

TL;DR

• —Base: Qwen3.8-27B (27B dense vision-language model)
• —Quant: GGUF Q4_0 mixed; symmetric 32-value blocks for the primary quantized tensors
• —Tool: auto-round-best from Intel AutoRound 0.14.2 (1000 iterations, 512 samples)
• —Size: 15.72 GB language-model GGUF plus 1.84 GB mmproj projector
• —Context: 262,144 tokens natively
• —MTP: One Multi-Token Prediction layer is retained in the language-model GGUF
• —Vision: Image input is supported through the included mmproj-model.gguf; video input depends on the llama.cpp build's video support
• —Calibration: 50% Open-SWE code traces, 25% OpenMathReasoning, and 25% OpenThoughts3 science examples

Quick inference with llama.cpp

Use a recent llama.cpp build with Qwen3.8 support. The GGUF already contains the tokenizer and chat template, so no separate tokenizer files are required.

Text and code

llama-cli \
  -m Qwen3.8-27B-Q4_0-AutoRound-Code.gguf \
  -ngl 999 \
  -c 262144 \
  -n -1 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  -cnv

The model's chat template enables thinking by default. Use llama.cpp's chat-template options when you need to change reasoning behavior, for example:

llama-cli \
  -m Qwen3.8-27B-Q4_0-AutoRound-Code.gguf \
  --chat-template-kwargs '{"enable_thinking":false}' \
  -p "Write a quicksort in Python."

Image input

Download both GGUF files from this repository and pass the projector with --mmproj:

llama-cli \
  -m Qwen3.8-27B-Q4_0-AutoRound-Code.gguf \
  --mmproj mmproj-model.gguf \
  --image path/to/image.jpg \
  -p "Describe this image."

For video input, use --video instead of --image with a llama.cpp build compiled with video support.

Quantization details

Tensor precision breakdown

The language-model GGUF contains:

• —505 Q4_0 tensors: 26,046,627,840 parameters
• —1 Q6_K tensor (output.weight): 1,271,398,400 parameters
• —360 F32 tensors: 2,671,616 parameters

The F32 tensors include normalization weights and state-space auxiliary tensors such as ssm_dt, ssm_a, ssm_conv1d, and ssm_norm. The separate vision projector is fully stored as F32.

Calibration data

The calibration file used for this quantization was mixed-code-math-science-512.jsonl, generated with seed 42 and pre-packed to approximately 2,304 tokens per row. AutoRound consumed it with seqlen=2048 and num=512; the pre-packed data was used without concatenating rows.

The SHA-256 of the exact calibration JSONL used was:

022cbb61e5c599a98c92f11304368f6352a2e1a575cffb47149e7f5ef15c5258

Reproduction

The quantization was run from the project workspace with the following command:

auto-round-best \
  --model ./models/Qwen3.8-27B \
  --scheme "GGUF:Q4_0" \
  --format "gguf:q4_0" \
  --dataset "./calib/mixed-code-math-science-512.jsonl:num=512" \
  --output_dir ./quants/Qwen3.8-27B-AutoRound-Best-Q4_0-Mixed \
  --seqlen 2048 \
  --nsamples 512 \
  --bs 1 \
  --gradient_accumulate_steps 8 \
  --low_gpu_mem_usage

The source model is `Qwen/Qwen3.8-27B`. The mmproj-model.gguf projector was exported from the same source model for llama.cpp multimodal inference.

Acknowledgements

• —Alibaba / Qwen team for the base Qwen3.8-27B model
• —Intel AutoRound for the quantization framework
• —llama.cpp for GGUF inference and multimodal support
• —The maintainers of Open-SWE-Traces, OpenMathReasoning, and OpenThoughts3-1.2M for the public calibration sources

License

Apache 2.0 - same as the Qwen3.8-27B base model.

Citation

If you use this quant, please cite the original Qwen3.8 release (see the base model card) and the AutoRound paper:

@misc{qwen38,
    title = {{Qwen3.8-Max}: A New Bar for Coding and Cowork},
    url = {https://qwen.ai/blog?id=qwen3.8},
    author = {{Qwen Team}},
    month = {August},
    year = {2026}
}

@article{cheng2023autoround,
  title   = {Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs},
  author  = {Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},
  journal = {arXiv preprint arXiv:2309.05516},
  year    = {2023}
}