Frosty40/Qwen-AgentWorld-35B-A3B-B70-Turbo-GGUF

Qwen-AgentWorld-35B-A3B — B70 Turbo (GGUF)

GGUF builds of [Qwen/Qwen-AgentWorld-35B-A3B](https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B) tuned and validated for fast local agent inference on the Intel Arc Pro B70 with llama.cpp (SYCL).

This repository ships the `Q5_K_M` agent-augmented quant — the best quality/throughput all-rounder from an extended B70 optimization campaign — together with the exact serving recipe, full throughput and quality benchmarks, and an optional speculative-decode route for deep structured contexts.

TL;DR — Q5_K_M-agentaug, 34.7B total / ~3B active MoE, 5.71 bpw, 24.7 GB. On a single Arc Pro B70 it sustains ~85 tok/s decode on short agent/tool turns and >1,200 tok/s prefill, holding ~16.5 tok/s decode at 73k context — and 150–224 tok/s aggregate across a 32-agent fleet (2.4–3.5× via concurrency). Quant is near-lossless to the BF16 source (see Quality). Apache-2.0.

One Shot Game Prompts: https://agentic-arcade.vercel.app/model-agentworld.html

Where the speed comes from

Three configs, same Q5KM weights, same f16 KV, same GPU:

1. 1.upstream — mainline llama.cpp, default flags
2. 2.up+flags — same mainline + Turbo flags (GGML_SYCL_DISABLE_DNN=1 -b 8192 -ub 4096)
3. 3.Turbo (this build) — same mainline + 3 fused-kernel commits (topk-MoE router fusion, gate-glue fusion, single-token expert-aggregate) + same flags

Summary: the tuned runtime configuration (-b 8192 -ub 4096, oneDNN GEMM disabled) delivers the prefill (1.2–1.7×) and fleet (~1.3–1.4×) gains and applies to any llama.cpp SYCL build. The Turbo build's fused decode kernels add a further +7–14% to single-stream decode. Output quality is unchanged — identical weights, lossless serving.

• —Prefill 1.2–1.7× and fleet ~1.3–1.4× come from the tuned runtime configuration, reproducible on stock upstream llama.cpp (the Turbo build contributes 1.00–1.01× on top).
• —Single-stream decode +7–14% at every depth (0.8k→129k) is delivered by the Turbo build's fused kernels.
• —Trade-offs: at 8–16 concurrent agents, fused decode performs within ~2% of the unfused path (build win 0.98×; neutral at 24+). -ub 4096 trades VRAM (larger compute buffer) for the prefill win.
• —Quality: identical weights, lossless.
• —Provenance: the MoE reorder is merged upstream (ggml-org/llama.cpp PR #24452); the fused kernels above are specific to this build.

Full breakdown: see "Performance breakdown" under Throughput, below.

Model details

SHA256 (agentworld-35b-a3b-Q5_K_M-agentaug.gguf): 27a4c510844330e4980c96bd0d80f1511bacc9557aa7b21f812a81eeebd5d7b4

Files

Other quants (Q4KM speed / Q6K quality / Q80 near-lossless reference) were produced during the campaign and can be added on request.

Throughput on the Intel Arc Pro B70

Measured end-to-end through the llama.cpp OpenAI-compatible server (/v1/chat/completions, real agent prompts), single B70, ship config below. temperature=0, 512 generated tokens per request.

Decode & prefill vs. context depth (averaged over the two well-calibrated prompt classes, structured tool/log/code and novel long-context synthesis):

Short agent/tool turns (decode-dominated, the common interactive case):

Full per-request matrix (4 classes × depth buckets): `throughput_default.tsv`. Prefill peaks above 1,200 tok/s around 24k context; decode degrades gracefully with depth (KV-bandwidth bound), staying usable (~16 tok/s) at 73k. Decode is memory-bandwidth bound on this MoE, so smaller quants do not decode meaningfully faster — Q5KM is the quality/throughput sweet spot.

Canonical llama-bench

Reproducible pp512 / tg128 at matched KV depths (ship config: -fa on, q8_0 KV, -r 3, build 0b98ca1b4):

(Pure-generation llama-bench figures run ~2% above the end-to-end server numbers above, which include chat-template + HTTP overhead — the two cross-validate cleanly.)

Concurrent serving (agent fleets)

One decode stream under-feeds the B70 (it is memory-bandwidth bound), so running parallel agents multiplies aggregate throughput. Serving with -np N parallel slots, realistic agent turns:

A single B70 sustains 150–224 tok/s aggregate across 32 concurrent agents (real short tool turns hit 223 tok/s) — a 2.4–3.5× total-throughput gain over one agent, ideal for autonomous/background agent fleets. -np 32 is the recommended production point (knee of the curve, comfortable VRAM headroom); -np 48 is the clean peak. Do not exceed `-np 48` — at -np 56 the 30 GiB is exceeded and throughput falls off a cliff (≈20× collapse). Concurrency trades against per-agent context depth (agents × context must fit 30 GiB), so reserve high -np for shallow, high-volume agent traffic and serve deep-context work at -np 1–2. See `BENCHMARKS.md` §6.

Performance breakdown — flags vs. build (AgentWorld-35B-A3B, f16 KV)

The figures above combine two independent effects: runtime flags (GGML_SYCL_DISABLE_DNN=1 -b 8192 -ub 4096) and this build's fusion commits. Isolated on the same Q5KM weights, same GPU: flags win = (2)/(1) (mainline llama.cpp, same flags) · build win = (3)/(2) (Turbo build, same flags).

Prefill t/s vs prompt tokens

Decode t/s vs context depth

Fleet aggregate decode t/s vs agents (2048+256 synthetic)

Summary: the tuned runtime configuration (-b 8192 -ub 4096, oneDNN GEMM disabled) delivers the prefill (1.2–1.7×) and fleet (~1.3–1.4×) gains and applies to any llama.cpp SYCL build. The Turbo build's fused decode kernels add a further +7–14% to single-stream decode at every depth. At 8–16 concurrent agents, fused decode performs within ~2% of the unfused path (build win 0.98×; neutral at 24+). Output quality is unchanged — identical weights, lossless serving. Measured on plain Q5KM; the shipped agentaug imatrix variant has the same quant type and tensor layout, so throughput is identical. Raw data: AgentWorld-35B-A3B-B70-Turbo.

Quality

Quantization fidelity vs. the high-precision source, plus held-out perplexity on agent-domain and generic text. PPL measured with llama-perplexity (-ngl 99, fp16 KV, 512-token windows).

Perplexity (lower = better; held-out sets, ~23k tokens each):

The agent-augmented imatrix lowers agent-domain PPL by −0.42% with no regression on generic text (−0.009%) — the calibration choice is domain-appropriate and free.

KL-divergence vs. `Q8_0` reference — how faithfully Q5KM reproduces the higher-precision model's output distribution (agent-domain text):

Q5KM tracks the Q8_0 reference very closely — mean KLD ~0.006, top-token agreement ~97%, and perplexity within 0.2%. (Measured over a time-boxed 4k-token agent-domain sample; the base-imatrix variant is statistically equivalent at Mean KLD 0.00671 / 97.45% same-top.) For reference-grade fidelity, the Q8_0 build is available on request.

The shipped quant uses an agent-augmented importance matrix: the base calibration corpus (generic) was blended ~31% with authored agent-domain text (tool-call XML, terminal/k8s/JSON/log/code/reasoning traces) so the Q5KM rounding preserves the tensors that matter most for agentic use. The effect is small but in the right direction and domain-appropriate (see PPL table; agent text improves, generic does not regress).

How to run (Intel Arc B70, llama.cpp SYCL)

Built from llama.cpp with the SYCL backend (oneAPI). The validated default-turbo route:

GGML_SYCL_DISABLE_DNN=1 ONEAPI_DEVICE_SELECTOR=level_zero:gpu \
  llama-server \
  -m agentworld-35b-a3b-Q5_K_M-agentaug.gguf \
  --alias agentworld-35b-a3b-turbo \
  -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -c 131072 -np 1 -b 8192 -ub 4096 \
  --host 127.0.0.1 --port 8091 --jinja

Why these flags:

• —GGML_SYCL_DISABLE_DNN=1 — native SYCL GEMM ≥ oneDNN for this MoE shape, and gives deterministic decode.
• —-ngl 99 — all 40 layers + experts fit the B70's 30.3 GiB at Q5KM with full 131k KV.
• —-fa on -ctk q8_0 -ctv q8_0 — flash-attention + q80 KV cache: large context within VRAM, q80 chosen over q40 (q40 was both slower and lower quality on this shape).
• —-b 8192 -ub 4096 — the prefill sweet spot (a large micro-batch is the dominant prefill win here).

Then call it as an OpenAI-compatible endpoint (/v1/chat/completions). It is a thinking model — it emits a <think> reasoning block before the answer.

Optional: structured-deep speculative route

For very large structured contexts (logs / code / tool output / observation echo) an n-gram speculative decoder gives a real speed-up at depth (it drafts from the repetitive structure). On the same model:

... llama-server ... \
  --spec-type ngram-map-k4v \
  --spec-ngram-map-k4v-size-n 8 --spec-ngram-map-k4v-size-m 24 \
  --spec-ngram-map-k4v-min-hits 1 --spec-draft-n-max 24

Measured +19% decode at 45k and +37% at 73k on structured incident-analysis prompts (byte-identical output to the non-spec baseline at 73k). It is opt-in / route-by-class: it does not help novel prose and is roughly neutral at shallow depth, so route only structured-deep traffic to it.

Recommended sampling

The base model's defaults (carried in the GGUF): temperature 0.6, top_p 0.95, top_k 20. Use temperature 0 for deterministic agent/eval runs.

Example outputs

Real generations from this quant on the B70 (temperature 0), illustrating the two core modes.

Tool-result → user-facing reply (decode-dominated agent turn, 83 tok/s):

User: get_weather returned {"location":"Austin, TX","temp_f":98,"humidity":0.41,"wind_mph":12,"conditions":"clear"}. Write the assistant reply. AgentWorld: The current weather in Austin, TX is clear with a temperature of 98°F. The humidity is at 41%, and the wind is blowing at 12 mph.

Structured incident triage → strict JSON (the structured-deep use case):

User: (kubectl pod CrashLoopBackOff + ERROR migration check failed code=E_SCHEMA_MISSING table=ledger_events + /healthz {"ready":false,"db":"ok","schema":"missing"}) — return JSON with root_cause, evidence, next_command. AgentWorld: ``json {"root_cause": "Database schema migration failed due to missing 'ledger_events' table, causing the payments-api pod to enter CrashLoopBackOff.", "evidence": "kubectl.logs shows 'ERROR migration check failed code=E_SCHEMA_MISSING table=ledger_events', and http.get /healthz returns {'ready':false,'db':'ok','schema':'missing'}.", "next_command": "Apply database migrations to create the missing 'ledger_events' table schema."} ``

It reasons inside a <think> block first, then emits the final answer (clean prose or strict JSON as asked).

Provenance & reproducibility

• —Source: Qwen/Qwen-AgentWorld-35B-A3B → convert_hf_to_gguf.py --no-mtp → BF16 GGUF → imatrix → Q5KM. The published checkpoint is language-model-only (no MTP/vision weights shipped), so it is converted with --no-mtp.
• —Importance matrix: agent-augmented calibration (~31% authored agent/tool/world-model text blended into a wide generic base corpus).
• —Engine: llama.cpp SYCL build for Battlemage (the Turbo build). The MoE mul_mat_id reorder is merged upstream (ggml-org/llama.cpp PR #24452); top-k-MoE router fusion + gated-delta-net gate fusion + expert-combine fusion are specific to this build (3 commits) and drive the single-stream decode gain — prefill/fleet throughput comes from the runtime flags alone, reproducible on stock upstream (see "Performance breakdown" under Throughput).
• —Hardware: 1× Intel Arc Pro B70 — 256 Xe cores, 30.3 GiB, 2800 MHz, 230 W cap (Level-Zero).
• —All benchmark scripts, raw TSVs, and per-request JSON are in this repository's release artifacts.

Limitations & caveats

• —Hardware focus: the recipe and numbers are tuned for the Intel Arc B70 (SYCL). It runs on any llama.cpp backend (CUDA/Metal/CPU), but the throughput figures are B70-specific.
• —Quant trade-off: Q5KM is near-lossless but not lossless; for reference-grade fidelity use Q8_0.
• —Thinking model: budget tokens for the <think> block; set a sufficient max_tokens.
• —Spec route is conditional: the n-gram route is a structured-deep optimization, not a global win.
• —Inherits base-model behavior and biases from Qwen/Qwen-AgentWorld-35B-A3B.

License & attribution

Released under Apache-2.0, inherited from the base model `Qwen/Qwen-AgentWorld-35B-A3B`. Please cite the original Qwen AgentWorld model. GGUF conversion, agent-augmented imatrix, and B70 tuning by Frosty40.