Kevletesteur/Qwen3.5-35B-A3B-Chimere-v3-GGUF

Qwen3.5-35B-A3B Chimere v3 -- RAMP GGUF

Chimere v3: Claude Opus 4.6 distillation of Qwen3.5-35B-A3B, optimized for instruction following and reasoning.

RAMP quantization (per-tensor quality overrides + imatrix) -- 15 GB, fits 16 GB VRAM, ~80 tok/s on RTX 5060 Ti.

Looking for v1 (best code + tools)? See Chimere v1 GGUF.

Compatible runtimes

This GGUF can be loaded by any runtime that supports the Qwen3.5-35B-A3B (qwen35moe) architecture. The reference runtime — and the one that exercises all chimere-specific features (Engram n-gram bias, multi-agent context switching, the C++ fast sampler with DRY + min-p, K-cache Hadamard rotation, fused MoE up/gate) — is chimere-server.

Benchmark Results

v3 strengths: instructions and reasoning

Qualitative agentic tests

Honest assessment

• —Strengths: 100% IFEval, 100% adversarial edge cases, 84% GSM8K, best overall reasoning
• —Weaknesses: Code generation slightly weaker (83% vs 97%), tool-calling lower (75% vs 90%)
• —Why: v3 dataset added IFEval-strict, OPSDC-compressed reasoning, and instruction-following samples on top of v1 base. Recommended for general agentic use.

Which version to use?

Best of both worlds: Use A-LoRA routing -- an intent classifier selects the appropriate LoRA at runtime. Code/tools queries use v1, instruction/reasoning queries use v3. See Chimere ODO.

Quick start (chimere-server, recommended)

# 1. Backend (one-time): build the ik_llama.cpp fork with sm_120 CUDA + Mamba-2 backport
git clone https://github.com/AIdevsmartdata/ik_llama.cpp.git ~/ik_llama.cpp
cd ~/ik_llama.cpp
git checkout mamba2-nemotron-h-backport
cmake -B build_sm120 -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 -DGGML_NATIVE=OFF
cmake --build build_sm120 -j

# 2. Server
git clone https://github.com/AIdevsmartdata/chimere.git
cd chimere/chimere-server
LD_LIBRARY_PATH=$HOME/ik_llama.cpp/build_sm120/ggml/src:$HOME/ik_llama.cpp/build_sm120/src:/usr/local/cuda-12.8/lib64 \
  cargo build --release --features server --bin chimere-server

# 3. Model + tokenizer
mkdir -p ~/models && cd ~/models
hf download Kevletesteur/Qwen3.5-35B-A3B-Chimere-v3-GGUF chimere-v3-ramp.gguf
hf download Qwen/Qwen3.5-35B-A3B tokenizer.json --local-dir tokenizers/qwen35

# 4. Run (production env vars)
CHIMERE_MODEL=$PWD/chimere-v3-ramp.gguf \
CHIMERE_TOKENIZER=$PWD/tokenizers/qwen35/tokenizer.json \
CHIMERE_LLAMA_BACKEND=1 \
CHIMERE_NCMOE=3 \
CHIMERE_KV_MAX_SEQ=65536 \
CHIMERE_PORT=8081 \
CHIMERE_FORCE_QWEN35=1 \
LD_LIBRARY_PATH=$HOME/ik_llama.cpp/build_sm120/ggml/src:$HOME/ik_llama.cpp/build_sm120/src:/usr/local/cuda-12.8/lib64 \
~/chimere/chimere-server/target/release/chimere-server

# 5. Hello world
curl -s http://localhost:8081/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'

Engram (optional, prod-only)

Chimere ships an n-gram logit bias overlay loaded from binary .engr tables. To enable it, set:

CHIMERE_ENGRAM_DIR=/path/to/engram_tables   # directory of *.engr files
CHIMERE_ENGRAM_ALPHA=0.1                     # logit bias strength

The engram tables are tokenizer-specific (Qwen3.5 vocab) and used as a per-domain overlay (kine, code, cyber, general). They are intended as a domain-knowledge injector, not a measured quality booster — see the chimere repo README for the honest status of the path.

Quick start (generic GGUF runtimes)

If you do not need the chimere stack, the GGUF works with any Qwen3.5-compatible runtime:

# llama.cpp / llama-server
llama-server \
    -m chimere-v3-ramp.gguf \
    -ngl 99 --n-cpu-moe 4 -c 32768 \
    --flash-attn on --jinja --port 8081

# For 16 GB VRAM (RTX 5060 Ti / RTX 4080):
# Add KV cache quantization to save VRAM:
# -ctk q8_0 -ctv q4_0

Recommended sampling parameters

Backend

The official chimere-server runtime links against a customized `ik_llama.cpp` fork (branch mamba2-nemotron-h-backport, head of upstream PR ikawrakow/ik_llama.cpp#1593).

Highlights of the chimere-specific layer on top of ik_llama:

• —Custom C++ fast sampler exporting sample_token_fast, set_logit_bias, set_engram_bias, clear_engram_bias and take_packed_logprobs — avoids a ~993 KB logits copy per token, packs OpenAI-format top-5 logprobs.
• —K-cache Hadamard rotation, fused MoE up/gate, grouped expert routing — all enabled by default via cparams.
• —Multi-agent KV / SSM state save & restore via llama_state_seq_*, keyed on the OpenAI user field. Up to CHIMERE_MAX_AGENTS (default 4) concurrent personas with their own conversation state.
• —An OpenAI-compatible HTTP layer in Rust (axum 0.8), supporting non-streaming and SSE streaming, tool calls, <think> reasoning extraction and chat_template_kwargs.enable_thinking.

Multi-architecture support

The same chimere-server runtime is not Qwen-only any more. As of Step 7 (April 2026), it dispatches between two code paths based on the GGUF's general.architecture metadata:

• —Qwen3.5-35B-A3B (qwen35moe) — full production stack: MTP, MRoPE, Engram, agent scheduler, custom Candle / cudarc / libllama paths. This GGUF.
• —Mamba-2 / Nemotron-H MoE / Mamba-1 / Mamba-2 hybrids — libllama-only path via GenericModel. No MTP, no Engram, single-agent only at Step 7. Validated end-to-end on unsloth/Nemotron-3-Nano-30B-A3B-GGUF (Q40 and UD-IQ3XXS) at ~45 tok/s on RTX 5060 Ti, NCMOE=30, ctx 2048, via the bundled test-nemotron smoke binary.

Models that should run via the same Generic path (untested at the chimere level — your mileage may vary): Granite 4.0 H-Tiny / H-Small / H-Micro, Falcon-H1 0.5B – 34B, Bamba-9B v1 / v2, state-spaces/mamba2-*, mistralai/Mamba-Codestral-7B-v0.1, AI21-Jamba-Reasoning-3B.

RAMP Quantization Details

Custom per-tensor quality overrides -- critical paths get higher precision. Overall: ~3.78 BPW.

• —imatrix: Generated on BF16 model (B200, 192 GB VRAM), 200 calibration chunks
• —Result: 15 GB with zero quality loss on agentic benchmarks vs BF16

Training Details

v3 dataset additions (on top of v1 base)

• —+50 IFEval strict (5 constraint categories)
• —+30 strict code (no markdown)
• —+30 code gen with thinking
• —+30 instruction following
• —+20 OPSDC-compressed reasoning (-64% tokens)
• —+15 multi-turn agentic

Limitations

• —MTP infrastructure present, gated. This GGUF carries an MTP (multi-token prediction) head — chimere-server detects it via n_nextn_layer = 1 and exposes the speculative-decoding infrastructure (mtp_scheduler.rs, MtpOp FFI). An early March bench on a previous build measured +49.5% token acceptance rate for the MTP draft path; that figure is not currently reproducible because bench_mtp.rs:104-167 has Benchmarks 2 and 5 hard-coded as SKIPPED with the comment crash in ik_llama MTP graph, KV cache issue for layer 41. Until that fix lands the 80 tok/s figure above is the non-MTP path. We will re-publish the MTP gain once the bench passes.
• —Engram is a domain-knowledge overlay, not a measured quality boost. The only saved engram eval in the chimere repo (benchmarks/engram_trained_eval.json) was run on GPT-2 + wikitext-2 and shows a −13.39% PPL regression on that out-of-distribution setup. No Qwen3.5-specific perplexity eval has been published yet. Engram is shipped as an optional per-domain n-gram bias (kine, code, cyber, general); qualitative use shows specialized vocabulary in responses (drainage bronchique postural, EMII, ...) on the kiné domain, but there is no quantitative claim attached to it today.
• —Multi-slot concurrent decoding via `ik_llama.cpp` is broken under heavy load (ik_llama multi-slot bug, slot 0 contamination of system prompts under contention). The chimere-server production deployment is single-slot. Stock llama-server does NOT have this bug if you need parallel slots.
• —Tool-calling sampler defaults: presence_penalty defaults to 0.0 — a previous default of 1.5 killed code generation and long reasoning blocks. See chimere-server source.

Files

Related

• —chimere -- Official Rust runtime (chimere-server) with Engram, MTP, multi-agent, multi-arch dispatch
• —ik_llama.cpp fork -- Backend with Mamba-2 + Nemotron-H backport (PR #1593)
• —Chimere v1 GGUF -- Best code + tools
• —BF16 full weights -- For re-quantization or fine-tuning
• —LoRA adapter -- For further training
• —Chimere ODO -- A-LoRA intent routing

Citation

@misc{chimere-v3-2026,
  title={Chimere v3: Claude Opus 4.6 Distillation of Qwen3.5-35B-A3B MoE for Instructions and Reasoning},
  author={Kevletesteur},
  year={2026},
  url={https://huggingface.co/Kevletesteur/Qwen3.5-35B-A3B-Chimere-v3-GGUF}
}