Modelpublic

petr567/GLM-5.3-Flash-AJ-IQ2-XXS-Strix-Halo-256K

sourceHugging Facemitupdated 24d agoView on Hugging Face

0likes

Model Card

GLM-5.3 Flash AJ IQ2_XXS — Strix Halo 256K profile

Reproducible llama.cpp/Vulkan deployment profile for the `GLM-5.3-Flash-AJ-IQ2_XXS` GGUF. This repository contains the launch profile and measurements; the model weights remain in the upstream repository.

Results

Tested on AMD Ryzen AI MAX+ 395 with Radeon 8060S (Strix Halo, gfx1151) and 128 GB unified memory.

Metric	Result
Context	262,144 tokens
Short decode, 256 generated tokens	12.89 tok/s
Long-context decode, 15,904-token prompt + 128 generated tokens	9.67 tok/s
Cold prefill, 15,904-token prompt	89.32 tok/s
Native tool roundtrip	PASS (1/1)
Tool arguments	typed integers
Tool result accepted	yes
Repeated tool call	no

Native tool calling works end to end: the model emitted add_numbers(a=17, b=25) once, accepted the result 42, and completed the answer. The profile is suitable for direct-tool agent and coding workflows.

Selected profile

Parameter	Value
Quant	AJ-IQ2_XXS, 87.35 GB
Context	262144
KV cache	Q80 / Q80
Batch / ubatch	1024 / 512
Threads / batch threads	16 / 16
Vulkan nodes per submit	100
Flash attention	enabled
MTP	disabled
Temperature / top-p	1.0 / 0.95

MTP was disabled in the selected profile because sustained decode was faster without it on this system.

Run

Requirements: Linux, Docker, a Vulkan-capable gfx1151 device, and enough unified/system memory for the 87.35 GB model plus the 256K KV cache.

bash

./scripts/download-model.sh ./models
./scripts/build-runtime.sh
MODEL_DIR="$PWD/models" ./scripts/run-vulkan.sh
curl http://127.0.0.1:8080/health
python3 scripts/tool-smoke.py

The OpenAI-compatible endpoint is available at http://127.0.0.1:8080/v1. Set PORT, MODEL_DIR, IMAGE, or CONTAINER_NAME to override the script defaults. To require authentication, set API_KEY_FILE to a readable file containing the API key.

Reproducibility

—Source GGUF revision: 07c62fcdeaf1c05d22bd123c3da8058a1b1e63e2
—Runtime source: `eauchs/llama.cpp`
—Runtime commit: c9ddd6821c93871c53741344d35d1e440d60d9ea
—Exact profile: `profiles/strix-halo-256k.json`
—Raw summarized measurements: `results/speed-summary.json`
—Tool check: `results/tool-calling-smoke.json`

The first GGUF shard is passed to llama-server; it discovers the second shard automatically. Both files are pinned and verified by SHA-256 during download.