drawthingsai/Qwen3.8-27B-GGUF

Qwen3.8-27B-GGUF

Author: Feiteng this work is sponsored by [Draw Things](https://drawthings.ai/).

The primary goal of this repository is to provide Qwen3.8-27B model quantizations below 4 bits per weight (bpw) in GGUF format.

The below-4-bpw target applies to language model weights. Companion vision encoder files are provided in the formats listed below, with their final quality metrics reported separately.

We also publish compact quantized vision encoders with high output fidelity, resolving GGUF quantization compatibility issues for the Qwen3.8-27B vision encoder. The Q4_K, Q5_K, and Q6_K files support inference with unmodified llama.cpp. Compared with our own BF16 export (888.01 MiB), our Q4_K file is 314.46 MiB, a 64.59% size reduction, with 0.99537 cosine similarity against the original Hugging Face FP32 vision outputs.

DT Checkpoints

Both DT checkpoints score 100.00% on AIME 2026 in the reported evaluation.

bpw vs. Score

[image]

The plot uses the bpw values indicated by the DT checkpoint names. Dashed lines show BF16 reference scores. Each benchmark is shown separately, with a shared score scale.

Benchmark Comparison

Scores are percentages; higher is better. AIME 2026, GPQA Diamond, and BFCL v4 (1k) report accuracy. IFEval reports prompt-level strict accuracy (prompt_level_strict).

Sizes are reported in GiB (1 GiB = 2^30 bytes). "Includes MTP" indicates whether the file contains multi-token prediction weights; it does not indicate whether MTP was enabled during evaluation. "Size without MTP" excludes those weights.

MTP

DT-bpw2.96-IQ3_XXS retains higher precision in its MTP component. With the maximum draft length set to 3, preliminary measurements show an overall draft acceptance rate of approximately 60%.

Vision Encoder

The files below include our own BF16 and FP16 exports alongside the four quantized formats. BF16 and FP16 files are prepared for upload; their download links will become available after upload.

Sizes are reported in MiB (1 MiB = 1,048,576 bytes). Lower MSE is better; higher cosine similarity and benchmark accuracy are better.

MSE and cosine similarity compare full vision outputs, including the merger, from llama.cpp CUDA against the original Hugging Face FP32 outputs. All six reported formats use the same evaluation conditions.

MSE is the total squared difference between the GGUF and reference vision encoder outputs (including the merger), divided by the total number of output elements across all evaluation images. Cosine similarity is computed over the flattened, concatenated outputs of those images. MSE and cosine similarity measure vision output fidelity.

RealWorldQA and OCRBench report end-to-end accuracy (higher is better), using the same Qwen3.8-27B-DT-IQ3_XXS.gguf language model with each vision encoder on llama.cpp CUDA. Each dataset has 100 fixed test examples; OCRBench includes 10 per category. These are quick-subset scores, not full benchmark results.

Thinking is disabled and responses are limited to 1,024 tokens. Responses reaching that limit remain included in the scores. RealWorldQA limit counts: BF16: 5; FP16: 2; Q80: 7; Q6K: 2; Q5K: 2; Q4K: 5. No OCRBench responses reached the limit.

All 1,200 examples have scored responses. FP16 required one server restart during OCRBench; completed examples were retained and the remaining examples were evaluated with the same configuration. One answer changes a dataset score by one percentage point.

Reproduce the Benchmarks

Reproduce the DT Results

Use profiles/qwen3.8-thinking.json to evaluate the DT checkpoints on AIME 2026, GPQA Diamond, IFEval, and BFCL v4 1K.

Clone the benchmark runner and install its evaluation dependencies in your Python 3.10+ environment:

git clone https://github.com/drawthingsai/benchmarks.git
cd benchmarks
python3 -m pip install 'evalscope[bfcl,ifeval]==1.11.0' 'soundfile==0.13.1' huggingface_hub

A compatible llama-server must be available on PATH; see the benchmark repository's setup instructions. Before running GPQA, accept the dataset access terms and authenticate:

hf auth login

The commands below use the final DT language model filenames and can be run once the files are uploaded to this Hugging Face repository. The mmproj files are companion vision encoders and cannot be used as the language model in these commands.

model_dir="$PWD/models/drawthingsai/Qwen3.8-27B-GGUF"
gguf_296='Qwen3.8-27B-DT-IQ3_XXS.gguf'
gguf_248='Qwen3.8-27B-DT-IQ2_S.gguf'

hf download drawthingsai/Qwen3.8-27B-GGUF \
  "$gguf_296" "$gguf_248" \
  --local-dir "$model_dir"

Run both evaluations from the cloned benchmarks directory, in the same shell:

python3 benchmark.py run \
  --profile profiles/qwen3.8-thinking.json \
  --gguf "$model_dir/$gguf_296" \
  --model-name DT-bpw2.96-IQ3_XXS \
  --run-id DT-bpw2.96-IQ3_XXS \
  --parallel 4

python3 benchmark.py run \
  --profile profiles/qwen3.8-thinking.json \
  --gguf "$model_dir/$gguf_248" \
  --model-name DT-bpw2.48-IQ2_S \
  --run-id DT-bpw2.48-IQ2_S \
  --parallel 4

python3 benchmark.py compare \
  runs/DT-bpw2.96-IQ3_XXS \
  runs/DT-bpw2.48-IQ2_S \
  --output results/dt-comparison.md

These commands evaluate all four language benchmarks with MTP speculative decoding disabled. Add --dry-run to a run command to inspect its configuration before evaluation, or --resume to continue an interrupted run with the same settings.

BFCL v4 1K evaluates 1,002 examples without Web Search; this is a subset score, not the official BFCL v4 Overall score. The current comparison profile replaces the former qwen3.8-thinking-bfcl-1k.json profile.

Reproduce the Community Baselines

See Reproduce the community baselines for pinned community model downloads and evaluation commands. Use the same profiles/qwen3.8-thinking.json profile and llama.cpp build as for the DT runs above.