malaiwah/Qwen-3.6-35B-A3B-SIQ-K6-K4K3

Qwen-3.6-35B-A3B-SIQ-K6+K4K3

A lightweight, structurally faithful dev proxy for the GLM-5.2 SIQ serving stack — Qwen3.6-35B-A3B (VL MoE) with routed experts quantized as mixed Trellis K3/K4 in the exact per-layer partition geometry of willfalco/GLM-5.2-EXL3-TR3-3.36bpw, served with online Trellis K6 conversion of eligible BF16 dense matrices on the vLLM gilded-gnosis + SparkInfer runtime.

This will not make Qwen3.6 any smarter — that is not the goal. The goal is a test model light enough to load in minutes on one 32 GB GPU, yet representative enough (same 256-expert tier space, same mixed-K runtime contract, same loader/kernel paths) that changes to vLLM-GG, SparkInfer (b12x) and the SIQ method can be smoke-tested without touching a GLM-5.2 production deployment or waiting for ~330 GiB of weights to load.

"SIQ" (SparkInfer Quantization) is the honest name for the format the community ships as "EXL3-TR3": it uses the ExLlamaV3 Trellis/MCG math (quantized with exllamav3 0.0.43's LDLQ pipeline) but the rank-sliced checkpoint layout requires the custom vLLM + SparkInfer runtime — it will not load in ExLlama, Transformers, or stock vLLM.

Why this model mimics GLM-5.2's SIQ deployment

Measured on AIBoss (RTX 5090 32 GB, r25 image, 2026-08-04)

Reference point: the GLM-5.2-EXL3-TR3-3.36bpw production stack this mimics runs 91.8 tok/s CC1 / 314.9 CC8 / 2.4k–2.0k tok/s prefill on 4× RTX PRO

1. 1.The proxy loads in seconds and iterates faster than prod — which is its entire purpose.

KLD vs BF16 (data point, not a benchmark)

Method: top-512 truncated KLD renormalized over the BF16 reference's support; 2,044 positions of the Apache-2.0 LICENSE text in 512-token chunks; prompt-logprob probes, offline vLLM r25, identical treatment for all three engines (tools/kld_probe.py / tools/kld_compute.py). A single fixed text — a data point, not a benchmark.

Ecosystem comparison & serving-path validations (2026-08-04)

• —Speed vs fidelity, same box/model: nvidia's NVFP4 build decodes 257.9 tok/s CC1 (fused NVFP4 MoE kernels via --moe-backend b12x, validated on this stack) vs this SIQ build's 158.7 tok/s — while SIQ measures 1.8× closer to BF16 (KLD 0.0105 vs 0.0186). Pick your axis.
• —Structured output under MTP (the vllm#34650 risk combo): 10-shot json_schema-constrained battery under speculation → 0 schema violations, 0 parse failures (baseline arm also 0 violations). No grammar bypass reproduced on r25 at k=2.
• —Vision tower stays BF16 by evidence, not laziness: every major quantized VL release (incl. NVIDIA's NVFP4 of this exact model and Qwen's own FP8 builds) excludes the tower; savings would be ~0.44 GB (~1.5% of a 32 GB card). If KV headroom ever demands it, weight-only MXFP8 via the online overlay is one ignore-list edit away — validate OCR/dense-text before trusting it.

Methodology notes (and a hardware war story)

• —Calibration: the GLM owner corpus (12,228 rows, 4 axes, sha-pinned) re-tokenized for Qwen; 1,052,436 tokens, natural top-8 routing, per-expert LDLQ Hessians; zero cold experts outside layer 0 (one, floor-fallback).
• —The build machine exhibited silent memory corruption (~1 bit per ~120 GB written; non-ECC DDR5, 4-DIMM XMP) during capture. Every stage of this pipeline is hash-gated (seal-time cache-bypassed re-read verification, encode-time re-hash, byte-verified assembly), so corrupted captures were quarantined and re-captured rather than encoded: nothing unverified reached this artifact. The MANIFEST.sha256 covers every shipped file.
• —Reproducibility contract: the derivation chain is fingerprint-verifiable (corpus → capture plan → per-layer manifests → k-plan → recipe), but re-derivation is functionally equivalent, not bit-identical — capture row order varies with engine scheduling and cross-process encode pooling is not bit-stable. Authenticity anchors on the published hashes.

V2: SIQ MTP layer (speculative-decoding test mode)

The MTP layer's 256 routed experts are ALSO quantized (mixed K3/K4, partition inherited from layer 0 as the runtime requires): the model-mtp shard is 0.38 GiB instead of 1.57 GiB. Calibration is a disclosed proxy: verified layer-39 activations routed through the MTP's own gate — sound for a draft layer, whose quantization error can only lower acceptance rate, never corrupt output (the target verifies every draft).

Speculative serving works (add to the serve command):

--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' \
--compilation-config '{"cudagraph_mode":"NONE"}' --max-num-seqs 16

Measured: mean acceptance length 2.35-2.38 / 3 (k=2) on RTX 5090. Two operator notes: (1) CUDA graphs MUST be off under speculation — the GDN spec path hits q_len_per_req is part of the frozen cudagraph shape (the same frozen-shape bug class documented on GLM-5.2 production, now reproduced on the qwen3_5 path — which is this proxy doing its job); (2) speculation is NOT a throughput win on a 3B-active model at TP1 eager (CC1 ~52 tok/s vs 158.7 without) — launch overhead dominates tiny per-step math. It exists to exercise the production draft path, and it does.

V3 findings: mixed-tier shared-memory ceiling on consumer Blackwell

An attempted layer-0 quality upgrade (K3/K4 -> K4/K6, then K4/K5) mapped a hard kernel boundary instead: the one-grid mixed-Trellis kernel's shared -memory footprint at this geometry follows

smembytes = 93,184 + 8,192 x ((klo + khi) - 7) [layout trellis3t256]

measured: (3,4) = 93,184 fits | (4,5) = 109,568 exceeds | (4,6) = 117,760 exceeds the RTX 5090's 101,376-byte opt-in limit. On 99 KB-smem consumer parts, (3,4) is the working ceiling; (3,5) computes to exactly 101,376 (boundary-exact, untested); K6 mixed tiers are B300-class (227 KB) territory. Encoding at K5/K6 works fine — serving is the constraint. Filed for the SparkInfer folks as tier-pair smem scaling data.

Also verified: the r25 loader does not consume non-empty keep_nvfp4 tier-bitmap entries (provenance metadata only) — a keep-NVFP4-within-SIQ hybrid needs loader-side support first.

How to run

Requires the gilded-gnosis r25 image + the 2-file qwen3_5 rank-sliced loader patch (in tools/patches/, submitted upstream):

podman run -d --name qwen36-siq --device nvidia.com/gpu=all --shm-size 16g \
  -p 8000:8000 -v <this-repo>:/model:ro -v jit-cache:/cache \
  -v <patched qwen3_5.py>:/opt/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5.py:ro \
  -v <patched qwen3_5_mtp.py>:/opt/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5_mtp.py:ro \
  -e VLLM_EXL3_ONLINE_TRELLIS_BITS=6 \
  -e VLLM_EXL3_ENCODER_SOURCE=/opt/exllamav3-python/exllamav3 \
  -e VLLM_EXL3_ONLINE_CACHE_DIR=/cache/exl3-online \
  docker.io/voipmonitor/vllm:gilded-gnosis-v20-vllmf5981f1-si978cdb3-fi801d57a-cu132-20260803-r25 \
  vllm serve /model --served-model-name qwen-3.6 --port 8000 \
    --kv-cache-dtype fp8 \
    --quantization-config '{"linear":{"weight":"mxfp8"},"shared_experts":{"weight":"mxfp8"},"ignore":["re:.*\\.mlp\\.gate$","re:.*linear_attn.*","re:.*shared_expert_gate.*","re:.*conv1d.*","lm_head","re:.*embed_tokens.*","re:.*visual.*","re:mtp\\..*"]}'

The GDN ignore entries are load-bearing: Qwen's Gated DeltaNet layers contain micro-projections (in_proj_ba, N=64; shared_expert_gate, N=1) too narrow for the online-K6 encoder (128-alignment) AND the MXFP8 fallback (mm_mxfp8 needs N>=128) — they must stay BF16 or the engine fails at memory profiling. Two matching subtleties: fused modules are ignore-matched against their UNFUSED shard names (so in_proj_ba$ never matches — use the blanket re:.*linear_attn.*), and shared-expert projections bypass the ignore list entirely (governed by the shared_experts spec).

Notes: quantization auto-selects exl3 from the top-level hybrid_tr3_tail config block; runtime TP must equal the checkpoint's tp: 1; the rank-sliced path is CUDA-graph-safe (no --enforce-eager needed); do not enable --speculative-config (the MTP experts are BF16 in this build). First start builds the online-K6 cache under /cache/exl3-online; warm restarts hit it.

Reproduction — everything is in tools/

The complete, reviewed toolchain that produced this checkpoint (fork of the GLM-5.2 calibration_encoder bundle, single-GPU sm_120 edition):

• —capture_qwen36.py — TP1 BF16 calibration capture (vLLM cpu_offload_gb streams the 67 GiB source through 32 GiB of VRAM; softmax top-8 router recompute; 5×8-layer tmpfs windows; 1,052,436-token owner corpus).
• —encode_qwen36.py — adapter over the byte-pinned production encoder encode_tr3_v31.py (sha256 e9a85a47…75032, LDLQ + MCG Trellis, exllamav3 0.0.43): fused-3D expert reader, per-expert K via uniform-K pools, VL-preserving assembly.
• —k_plan_qwen36.py — router-mass-weighted K4 selection (top-N by routed token count per layer), sealed plan + tier_bitmap.
• —preflight_qwen36.py, bootstrap_ext_sm120.py, convert_qwen36.sh, run_campaign_qwen36.sh — gates, sm_120 quantizer-ext build, orchestration.
• —patches/0001-qwen3_5-rank-sliced-exl3-loading.patch — the 2-file vLLM-GG loader patch (rank-slice name normalization for the qwen3_5 family).

Pipeline gates: adversarial code review, per-K encoder smoke (lockstep == sequential byte-identity), --verify-engine router-recompute audit on captured data, per-K byte-equality oracle, fingerprinted resume at every stage, byte-exact carried-tensor verification at assembly.

Credits — prior art this stands on

• —Qwen team — Qwen3.6-35B-A3B (Apache-2.0).
• —turboderp — ExLlamaV3: the Trellis/MCG quantization math and the 0.0.43 quantizer ops this pipeline drives.
• —brandonmusic — GLM-5.2-EXL3-TR3-3.0bpw and the published calibration_encoder bundle (capture/encode/bootstrap toolchain + the owner calibration corpus) this fork derives from.
• —willfalco — GLM-5.2-EXL3-TR3-3.36bpw: the mixed K3/K4 donor-patch design and the partition geometry replicated here.
• —malaiwah — GLM-5.2-EXL3-TR3-MTP78 (MTP-layer Trellis overlay) and SparkInfer PR #114, the original runtime-dynamic mixed-Trellis implementation.
• —voipmonitor / local-inference-lab — the vLLM gilded-gnosis fork, SparkInfer (b12x) SM120 kernels, PR #117 (runtime-dynamic expert counts), and the r25 release this targets.
• —RTX6kPRO community — the collective GLM-5.2 SIQ serving work that defined the recipe (a16 experts-low/dense-high split, online-K6, fp8 KV).

Limitations

• —Requires the custom vLLM-GG + SparkInfer runtime (r25+) with the included qwen3_5 loader patch until it lands upstream. Not loadable anywhere else.
• —MTP experts are BF16: no speculative decoding with this build (V2 planned).
• —Quantization quality is representative, not tuned: calibration reuses the GLM owner corpus unchanged, and the K4 selection policy is router-mass weighted (the published selection policy of the GLM 3.36 build is private).
• —Vision tower is BF16 and untouched; vision quality should match the base.