klee100/Qwen3.8-Flash-Next-Uncensored-AutoRound-3bpw-MTP

Qwen3.8-Flash-Next-Uncensored AutoRound ~3 bpw

Base model: orcarouter/Qwen3.8-Flash-Next-Uncensored — OrcaRouter's BF16 abliterated (refusal-removed) build of the official Qwen/Qwen3.8-Flash-Next. So the lineage is: Qwen (official) → OrcaRouter (uncensored fine-tune) → this AutoRound quantization, pinned to OrcaRouter source revision 8336e613ea508b13c2159bd0f68965d97a606b95.

This is a coding-calibrated mixed-bit quantization. It retains the original vision tower, tokenizer, and BF16 PLE table. ~3 bpw applies to the routed experts, not the entire checkpoint; the PLE table alone is approximately 102.4 GB.

Checkpoints: · This model · Official-Qwen variant · GitHub: full vLLM patch and setup guide

The MTP head in this revision is copied byte-for-byte from the separate official-Qwen quantized release. Its 1,536 expert projections are W4A16G128, seven dense projections are W8A16G128, and remaining MTP paths are BF16. The backbone is derived from OrcaRouter, so the MTP draft and backbone have different source weights. Speculative-draft acceptance and serving behavior after this substitution have not been measured.

Chat template

chat_template.jinja and the embedded tokenizer_config.json template both use froggeric's Qwen Fixed Chat Templates v22.5 (855bffc49448e299789730ff92c9b8d834d6cc14, Apache-2.0). This replaces the source model's chat template. Earlier calibration and evaluation used the source template; the results below have not been rerun with this revision.

Quantization

Intel AutoRound SignRoundV2 tuned all 48 backbone MoE blocks using 64 repository-distinct coding/tool windows of 2,048 tokens and 50 iterations. Blocks 0–11 and 36–47 use W3A16G128; blocks 12–35 use W2A16G64. Dense attention and DeltaNet projections use W8A16G128; routers, shared experts, hyperconnections, vision, and PLE remain BF16. The calibration source is NVIDIA Open-SWE-Traces at f8fb5b3d2c787f85f8a00f5fe04fe3f1a11088ef (CC BY 4.0). See RECIPE.md and recipe.json for reproducibility details.

Validation

• —Structural audit: 35 indexed weight shards, 227,702 tensors, 152,981,659,128 tensor bytes. The quantized MTP tensors are copied exactly from the donor release; the other tensors retain their original storage.
• —The following generation and quality results were measured before the MTP substitution, with the OrcaRouter-native BF16 MTP head. They are historical results for the earlier repository revision, not validation of the current quantized-MTP checkpoint. Patched-vLLM generation and native MTP speculative decoding were tested on that earlier revision. A 131,072-token, one-token generation smoke test peaked at 64,927 MiB (~63.4 GiB) GPU memory on an RTX PRO 6000 Blackwell with a 0.61 utilization cap.
• —Paired held-out coding-trace BF16→quantized fidelity, one window and 128 scored positions per length: 8K context KL 0.03718 nats, top-1 agreement 96.09%; 32K context KL 0.13202 nats, top-1 agreement 92.19%.
• —Four-task mini-SWE-agent/SWE-bench Verified check: 4/4 resolved by the official grader. Testing used thinking mode with temperature 1.0, top-p 0.95, top-k 20, min-p 0, presence penalty 0, and repetition penalty 1. The per-turn max_tokens limit was 8,192; observed completions stayed below it. All task repositories were absent from the calibration shard.

Serving

Use the pinned patched vLLM runtime and guide, including its PLE SSD-offload and mixed-bit patch against vLLM a5a30471ff2bb7f0824f2da10e358af98d304472. Set QWEN_MODEL_DIR to this checkpoint and follow the guide's build steps; do not download the previous checkpoint. The PLE table stays on SSD. For the quantized MTP draft, use the patch's W4 MTP support. The current hybrid checkpoint needs a new end-to-end serving and quality test. Qwen's recommended thinking-mode sampling was used for the agent validation.

This derivative retains the Qwen Community License 1.0 notice.