Modelpublic

jcbtc/Gemma4-26B-CIRU-AMD-STRIX-vLLM

sourceHugging Faceapache-2.0updated 2d agoView on Hugging Face

0likes196downloads

Model Card

[image]

Gemma 4 26B Ciru AMD Strix vLLM

A 26B-total / 4B-active Gemma 4 build for high-concurrency local inference on AMD Ryzen AI MAX+ 395. This repository contains the actual group-64 symmetric INT4 compressed-tensors checkpoint and its matching BF16 MTP assistant, rather than a GGUF or a runtime-only profile. The quantized linear weights are W4A16; vision, router, and other excluded tensors remain at their original precision. The measured configuration uses vLLM, ROCm, AITER unified attention, and four-token MTP speculation.

The point of this build is useful work at many simultaneous request slots. A single Strix Halo served 32 executing requests in our speed tests. In the 64-app live showcase, two Halo systems each served 32 independent app-building agents. The 650+ tokens/s shown in the image is the combined two-system first-pass average, not single-system throughput.

What is in the release

Component	Format	Notes
Main model	Hugging Face safetensors, compressed-tensors `pack-quantized`	Symmetric group-64 INT4 for attention, MLP, and experts; BF16 activations; unquantized `lm_head`, routers, and vision tensors.
MTP assistant	`mtp/model.safetensors`, BF16	Matching Google QAT assistant, used at speculative depth 4.
Tokenizer and processor	Original Gemma 4 files	Preserved alongside the sharded model.

The INT4 data plus BF16 group scales cost *4.25 bits per quantized weight. That is not an overall bits-per-weight figure: the repository also has BF16 tensors and a separate BF16 draft. Main-model safetensors total about 15.66 GB (decimal); the assistant safetensors is about 0.84 GB*. This is a quantized conversion of Google's instruction-tuned QAT checkpoint, not a new fine-tune.

Benchmarks

These are observed output tokens divided by complete request-wave wall time, including prefill and the draining tail, unless noted otherwise. They are not isolated kernel or per-agent decode rates. Hardware is one AMD Ryzen AI MAX+ 395 with 128 GB unified memory for each single-host row; the exact prompt, cap, sampler, and runtime differ by row. Do not compare unlike rows as a controlled speedup.

Single-system test	Workload	Output rate	Other result
Warmed OpenAI-compatible API, Sozo	32 executing requests, 512-output cap, 16,384 output tokens	537.67 tok/s	32/32 completed; median streamed first token 0.947 s.
Sustained offline generation, Sozo	32 executing requests, 2,048-output cap, natural EOS, 59,914 output tokens	420.80 tok/s	Zero preemptions; 78.87% MTP draft acceptance.
EvalScope serving, Ciru	32 executing requests, 1,024 input + 2,048 forced output each, 65,536 output tokens	296.92 tok/s	32/32 succeeded; full five-second output peak 451.34 tok/s.
Cold 64K prefill, Ciru	65,504 input + 32 output, zero cache hits, two V3 runs	1,023–1,029 input tok/s	Time to first token 63.64–64.01 s; single request, not output tok/s.

The 64K prefill result uses our later V3 long-prefill kernel. The quality suites below ran before that kernel promotion; three exact long-context retrieval checks passed on both old and new kernels, but the full quality suite was not rerun after V3. A 262,144-token context was configured and boundary components were exercised; we do not claim 256K prefill at the 64K rate.

Quality

The complete native matched suite used its own deterministic evaluation protocol: GSM8K 95.45% (1,259/1,319), HumanEval 95.73% (157/164), and IFEval prompt-level strict 87.62% (541 prompts). All 2,024 requests were accounted for, with zero execution errors; five responses reached their output limit and remain in the scores. These are scores for this converted model and serving stack, not a claim of improvement over the Google source model.

A separate EvalScope scorecard used the recommended sampler (temperature 1.0, top-p 0.95, top-k 64, repetition penalty 1.0; seed 42; 8,192 output-token cap) and 64 queued clients behind 32 execution slots. Across 941 questions, sampled GSM8K scored 193/200 (96.5%), stratified sampled MMLU-Pro 145/200 (72.5%), and full IFEval 87.99% prompt-level strict. All three evaluations completed with zero execution errors. The GSM8K and IFEval scores in the two protocols should not be collapsed into a single result.

More exact settings and definitions are in the benchmark record.

See all 64 apps

**Open the dual-Strix-Halo 64-app showcase →**

The original generation pass ran 64 independent app requests (32 slots on each machine) in 5 minutes 32 seconds, emitted 219,988 output tokens, and averaged 661.7 combined output tok/s. The highest complete five-second window was 1,166 combined output tok/s. Sixty-three first-pass requests ended normally and one hit its output limit. The linked gallery contains the final 64 playable pages after subsequent repairs and small publication fixes; the speed and time figures exclude that later work. The two hosts averaged about 339 and 323 output tok/s over the shared wall time. This is an end-to-end workflow on two systems, not a one-machine model benchmark.

AMD provided one of the Ryzen AI Halo systems used to create the showcase content. The measurements and model conversion are Ciru's work.

Run with vLLM

Download the full repository so the mtp/ folder is present. With a Gemma 4-capable vLLM/ROCm build and sufficient available memory:

bash

hf download jcbtc/Gemma4-26B-CIRU-AMD-STRIX-vLLM --local-dir ./gemma4-ciru-strix

vllm serve ./gemma4-ciru-strix \
  --language-model-only \
  --max-model-len 8192 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 2048 \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","model":"./gemma4-ciru-strix/mtp","num_speculative_tokens":4,"quantization":null,"draft_tensor_parallel_size":1}'

This is a portable starting profile, not the measured 32-slot AMD configuration. The benchmarked systems used a Ciru-modified ROCm vLLM stack, --max-num-seqs 32, --max-num-batched-tokens 2048, --attention-backend ROCM_AITER_UNIFIED_ATTN, Triton MoE, an auto-selected dense W4A16 kernel, eager execution, and an explicit 64–80 GiB KV reservation depending on the test. Ciru's V3 attention patch was used for the final 64K prefill figure. The custom dense and prefill optimizations are not included in these weights, and unmodified vLLM has not been qualified for the listed Strix Halo speeds. For installation and API examples, see the official vLLM Gemma 4 guide. The measured application and quality runs were text-only (--language-model-only); the retained vision tensors are not a vision-quality claim for this release.

Google's generation defaults are temperature 1.0, top-p 0.95, and top-k 64. Set request parameters deliberately when comparing scores. The source model's chat template supports Gemma 4 thinking and tool calling; prompt formatting and parser support depend on the serving build.

Provenance and credit

—Main source: Google Gemma 4 26B-A4B IT QAT Q4_0 unquantized, revision f1e06dc520982d9b9edd76859fdb7ab209449949.
—MTP source: matching Google assistant, revision 9537141506fe8875b3ed45b264af13580cb29166.
—Quantization and AMD serving optimization: Ciru / John Crown. Inference framework: vLLM; AMD GPU kernels: AITER. Evaluation used EvalScope and the named benchmark datasets.

The upstream source models are published under Apache 2.0; this conversion is released under the same license. Please retain the Google and other upstream attributions when redistributing it.