PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

The balanced Qwen3.8-27B quant for AMD Strix Halo (Ryzen AI Max+ 395 / gfx1151). Measured, not marketed: one 15 GB file that serves 128K of context on the iGPU — full attention, no offload flags, speculative decoding built in.

Brand: PaoAI · community quant, not affiliated with Alibaba or the Qwen team.

Updated 2026-09-20 — re-swept on the latest engine, weights unchanged. Every depth point re-measured on the latest version (paoai-strix-engine strix-main @ 4905ff417): deeper than ever — the full 256K window served and integrity-checked for the first time (261,519 tokens) — with decode faster at every depth from 32K up. The previous version's numbers stay on this card labeled as such. See Update log.

What is this? (30 seconds)

Qwen3.8-27B is a hybrid: 48 cheap linear-attention layers and only 16 full-attention layers that actually re-read your context. We compressed it organ by organ — precision where it matters (attention K/V, the MTP draft head, every norm), 4-bit FP4 where it doesn't. One 15 GB file, no offload flags, speculative decoding built in, intact from 8K to 128K of filled context.

The recipe (what's inside)

Measured tensor sizes: FFN gate/up 6.16 GB · FFN down 3.08 GB · linear-attn qkv 1.42 GB · GDN state ops 0.81 GB · token embeddings 1.04 GB · output 0.68 GB · full-attn 0.62 GB · draft head 0.10 GB · norms <0.01 GB → 15.01 GB total.

Chain-test results (N=3, graded by execution)

We run a fixed coding exam where tasks are graded by execution, not opinion. Three identical runs per engine; we report medians.

A same-session reference run of the published engine also scored 100/100/100 · task 100. Zero protocol events on either engine — quality is a straight tie.

Speed (decode, tokens/second, per phase)

The published column is from September. The defensible comparison is the last column — same session, same flags, engine the only variable: +6 % to +16 %.

Does quantization hurt? (perplexity + KL test — measured 2026-09-17, published 2026-09-18)

Plain words: the chain test above can only tell you if the model still passes. This test asks a different question: did compressing the weights change what the model actually predicts? We ran the industry-standard check — llama.cpp's own llama-perplexity tool — comparing this 15 GB FP4 file against a full-precision F16 copy of the same model (built by us from Qwen's official BF16 release, sha-verified shards), reading the same ~295,000 tokens of standard Wikipedia text through both at 8K windows.

Plain read: on standard text, the FP4 cut is statistically indistinguishable from the full-precision model — no measurable quality loss. The small difference that exists is concentrated in a tail of rare positions (~1 % of tokens shift noticeably), and the chain test above shows those shifts didn't cost task success. Not lossless in the mathematical sense — lossy in a way we cannot measure on standard text.

Reproducible: llama.cpp `llama-perplexity`, engine commit `11bfe8a`, wikitext-2-raw, `-c 8192`, 36 chunks (~295K tokens). Reference F16 converted by us from Qwen's official BF16 release (sha-verified shards). Full data in our results file.

Context decay — what happens when the window fills (8K → 256K)

[image]

Latest version (2026-09-20, `4905ff417`): fresh prompt and fresh server per depth, default batching. Previous-version columns (2026-09-17, fork `6d1d80e3f`) from the same protocol — comparable.

Deep range note (new, measured): past ~200K the default batching can lose the Vulkan device (vk::DeviceLostError, reproduced at ~213K). On the latest version the fix is the same small-batch rule the Flash-Next card uses: serve the deep range with -ub 256 — verified clean to 261,519 tokens (integrity ✓, acceptance 0.84). The auto-switch in the engine handles this automatically when serving with wide batches; at default batching, add -ub 256 for 200K+ work.

Plain read: decode is faster on the latest version at every depth from 32K up (+15–27% at 96K–128K), prefill catches up and wins from 64K (+6–8%), and the window now honestly reaches 256K with integrity — previously only reached 192K, and never on the card.

Facts

Requirements — which llama.cpp build

This model uses two things stock llama.cpp doesn't have: the qwen3_5 architecture (hybrid linear + full attention + MTP) and the Q4_0_ROCMFP4_STRIX quant type (type 105, gfx1151-native).

• —Recommended (latest version): guevae2/paoai-strix-engine, branch strix-main @ 4905ff417 — every number labeled "latest" on this card
• —Previous version: kingjones30/ROCmFPX, branch z13-main, commit dfeacaf2f — the build the original card measured.

How to run (measured, locked)

llama-server -m Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED.gguf \
  --host 0.0.0.0 --port 8091 -ngl 99 -c 65536 \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  -fa on --cache-type-k q8_0 --cache-type-v q8_0 --reasoning-budget 1500

Same flags for both engines. Raise -c for long context (up to 262144). Do not add -b/-ub 2048, and do not enable adaptive drafting — both were measured and rejected (see Known limits).

Thinking budget — --reasoning-budget caps how long the model thinks before answering:

Known limits (honest)

• —Depth costs speed: prefill 306 → 114 t/s and decode 31 → 14 t/s from 8K → 128K of filled context. The 16 full-attention layers re-read history for every token; the 48 linear layers stay cheap. Linear, no cliff — but plan for it at 100K+.
• —Decode is not a clean win on the new engine: better at 8K–64K, worse at 96K (14.0 vs 16.5), level at 128K.
• —Caveat on the comparison: the "prev" numbers were measured under an unrecorded power state; the fork's are at the AC performance profile — so the deltas blend engine + profile. The clean single-variable figure is +18–27 % prefill (4-arm test, identical fills, same-session reference).
• —Not adopted, and why: adaptive drafting (1 formatting failure in 3 chain runs) · wide batch -b 2048 -ub 2048 (no gain, and it lost the GPU device at depth).
• —Vulkan only: every number here is from the gfx1151 Vulkan backend on a Strix Halo iGPU. CPU builds and other GPUs are not comparable.
• —Long thinking: a bigger --reasoning-budget means noticeably longer waits before the answer starts.
• —Don't crank context blindly: a single 15 GB file, but GTT grows with context (≈51.6 GB at 64K). Raise -c in steps on a 128 GB box; don't merge shards.

Serving features explained (plain words)

Draft acceptance on the depth-sweep probes measured 0.58–0.83 on the fork — lower than the 0.86–0.90 on the original card. The sweep prompts are repetitive filler, which a draft head predicts poorly, so acceptance there is not comparable across tests. On the chain (real work) the fork decoded faster than the previous engine.

Update log

Earlier engine numbers are not repeated here — they were measured on a different build; the tables above are the current measurement. If you reproduce a number, quote the engine commit with it.

Fleet recipes

Serving configs and our benchmark battery: [paoai-strix-recipes](https://github.com/guevae2/paoai-strix-recipes) (GitHub).

Credits & license

• —Qwen team (Alibaba) — Qwen3.8-27B base model (Apache-2.0)
• —ggml-org / llama.cpp contributors — the engine, the quant formats, the Vulkan backend
• —LaurentZuijdwijk — the Strix-Halo-tuned Vulkan fork measured on this card
• —nathanw1014 — ROCmFP4 types, qwen3.5/MTP work, Strix Halo gfx1151 performance
• —kingjones30 / charlie12345 — ROCmFPX format & recipes
• —vcruz305 — converter branch work (qwen3_5 support path)
• —unsloth — imatrix methodology reference
• —deepu105 — same-box Strix Halo proof and serving-config intel
• —antirez — GGUF layout reference
• —PaoAI — the STRIX recipe (per-tensor cut/keep table), imatrix-guided builds, sweep benchmarking, chain test

Apache-2.0, same as the base model. Not affiliated with Alibaba or the Qwen team.