axjns/strix-halo-inference-bench

Strix Halo Local Inference Benchmarks

Measured prefill and decode throughput, and real VRAM cost, for local GGUF models on AMD Strix Halo (Radeon 8060S / gfx1151) under ROCm.

Why this exists

Strix Halo inverts the usual local-inference trade-off. A discrete 24 GB card gives you high memory bandwidth and a hard capacity ceiling; Strix Halo gives you the opposite — up to 64 GiB addressable as VRAM out of 128 GB unified, at substantially lower bandwidth. That changes which models are worth running, and by how much.

Almost none of this is documented with numbers. People evaluating these machines are working from forum anecdotes and vendor slides. This dataset is an attempt to fix that for one specific, fully-specified machine, with the harness published so the numbers can be checked.

The headline result is not about quantization at all — it is about architecture. Decode throughput, every model measured, same machine:

Every MoE beats every dense model by roughly 5-7x on decode, and a 120B MoE at 58.9 GiB decodes faster than a 27B dense model at 16.9 GiB. On memory-bandwidth-bound hardware the active parameter count sets decode speed, not the total — so unified memory's large capacity is worth most when spent on sparse models.

Quantization matters too, but an order of magnitude less. Qwen3.6-27B, same model and machine, across four quantizations:

Decode throughput falls steadily as the weights grow; prefill stays flat within noise. That is the whole platform in one table: prefill is compute-bound and this iGPU handles it respectably, decode is memory-bandwidth-bound and that is where you pay. *On this hardware the quantization you choose is your decode speed, and prefill numbers will not warn you about it.*

Every model measured so far:

Any capacity planning that quotes a single "tokens/sec" figure for this hardware is hiding the number that actually matters for interactive use.

What's measured

One row per (model, quant, benchmark point). Throughput comes from llama-bench; VRAM is sampled independently because llama-bench does not report it.

lb_* fields are passed through rather than re-derived, so the schema survives upstream llama-bench changes and you can always recover the original output.

Method

• —llama-bench with -ngl 999 (all layers on GPU) unless llama_bench_flags says otherwise.
• —3 repetitions per point; mean and stddev both recorded.
• —Four points per model: prefill at 512 and 4096, decode at 128 both cold and with a 4096-token KV cache already warm. Decode-at-depth is included because decode throughput degrades with context and single-number benchmarks hide it.
• —VRAM sampled from rocm-smi at 4 Hz throughout the run; peak retained.
• —Baseline VRAM captured immediately before launch so model_vram_gib reflects the model, not whatever else was resident.

Caveats — read before citing

• —`gpu_contended=true` rows are not comparable to clean rows. Throughput is depressed and baseline VRAM is inflated. They are kept, and flagged, rather than quietly dropped.
• —Unified memory means "VRAM" is a soft boundary. The 64 GiB figure is the current allocation out of 128 GB, not a hardware limit; results shift if you reallocate.
• —One machine, one build. These numbers characterise this box at this ROCm and llama.cpp version. Both are recorded per row; do not generalise across versions.
• —Quantization is parsed from filenames. Convention-dependent, and unknown where the filename doesn't say.
• —Two models are absent, not omitted. diffusiongemma-26B-A4B-it (a diffusion LM) fails to load in this llama.cpp build, and DeepSeek-V4-Flash-ROCmFP3 (95 GiB) exceeds the 64 GiB VRAM allocation. Both are real limits worth knowing.
• —Throughput is measured, quality is not. A fast quant that degrades output is not a good quant, and nothing here tells you which is which.

Reproducing

python strix_bench.py --list                 # discover local GGUFs
python strix_bench.py --all --dry-run        # show the plan
python strix_bench.py --model Qwen3.6-27B    # a quant sweep of one model
python strix_bench.py --all                 # everything

The harness refuses to run if it detects another GPU workload, since that would silently corrupt the numbers. --allow-contended overrides and tags the rows.

Scope

Deliberately narrow: one machine, fully specified, numbers you can reproduce. It is not a leaderboard and makes no claim about which model is "best" — only what each costs to run here.