xero0000/Alyx-AgentWorld-35B-A3B-mixed-q2k

🤝 Alyx — Qwen-AgentWorld-35B-A3B Mixed q2_K + imatrix

Black Mesa mixed-quant series · the agentic companion.

A CPU-offload-aware mixed-precision GGUF of Qwen-AgentWorld-35B-A3B — the official Qwen "language world model", trained to simulate agent environments (MCP/tool-calling, Search, Terminal, SWE, Android, Web, OS). Runs the full 256K context at ~79 tok/s decode on a single 18 GB dual-GPU desktop, by quantizing the CPU-offloaded expert layers to Q2_K while keeping the GPU-resident tensors at Q4_K.

TL;DR: an ~18 GB file that runs a Claude-Sonnet-class agentic model (AgentWorldBench 56.4) at full 256K context, at small-context speed — the daily driver for browser/tool/OS agent loops.

What it is

**Qwen-AgentWorld-35B-A3B** is Qwen's agent-environment world model on the hybrid qwen35moe architecture (gated attention

• —gated-delta-net SSM, 256 experts, 8+1 active, ~3B active/token). It is thinking-ON by design — it reasons about environment state in <think> before acting — and scores 56.4 on AgentWorldBench, roughly Claude-Sonnet-class on agentic tasks.

This build applies the mixed q2_K + imatrix recipe so you get that model at the fast 256K profile on limited VRAM. GGUF lineage: quantized from DevQuasar's Q8_0 conversion (the original safetensors use Qwen3_5MoeForConditionalGeneration, which llama.cpp's converter doesn't handle directly).

Recipe

• —Mixed layout:
• —ffn_*_exps on blocks 13–26 → `Q2_K` with importance matrix (the CPU-offloaded set)
• —everything else → `Q4_K` · output-class → `Q6_K`
• —~4.9 bpw effective, ~18 GB, 256K native context.
• —Why it works: decode on CPU-offload rigs is RAM-bandwidth-bound, so only the offloaded-layer bytes matter — those get Q2_K while GPU-resident tensors keep Q4_K quality. See the Gordon card for the full rationale.

Measured performance

Rig: RTX 3060 Ti 8 GB + RTX 3080 10 GB, DDR4, ik_llama.cpp:

Want more speed? **Alyx-MTP** is this same file with a transplanted multi-token-prediction head for self-speculative decoding in ik_llama.cpp (~79-82 tok/s at 128K).

How to run

Works in llama.cpp and ik_llama.cpp. The tuned 18 GB dual-GPU profile (ik):

./llama-server -m Qwen-AgentWorld-35B-A3B-mixed-q2k.gguf \
  --jinja --cache-type-k q4_0 --cache-type-v q4_0 --flash-attn on \
  --ctx-size 262144 --parallel 1 --n-gpu-layers 99 --ctx-checkpoints 8 \
  -ot 'blk\.(1[3-9]|2[0-6])\.ffn_(up|gate|down)_exps\.weight=CPU' \
  --tensor-split 44,56 --ubatch-size 256 \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  --no-mmap --threads 8 --no-warmup

• —Do NOT disable reasoning — the <think> environment-state reasoning is the point of this model. Give it generous max_tokens.
• —Recommended sampling (per the base model): temp 0.6, top-p 0.95, top-k 20.
• —--ctx-checkpoints 8 caps ik's dynamic SSM checkpoints (default 32 × 64 MiB ≈ 2 GB at deep context — an OOM trap on long agent sessions).

Credits

• —Base model: Qwen/Qwen-AgentWorld-35B-A3B (Apache-2.0)
• —Q8_0 GGUF source: DevQuasar
• —Mixed quant, imatrix & profiling: xero0000
• —Series: Gordon (base) · Kleiner (coder) · G-Man (uncensored) · Vortigaunt (reasoner) · Alyx (agentic)