davetha/Qwen3.8-Flash-Next-DERISKED-W4A16-AWQ

Qwen3.8-Flash-Next-DERISKED-W4A16-AWQ

A W4A16 (AWQ) quantization of `Blackfrost-AI/Qwen3.8-Flash-Next-DERISKED-BF16`, an abliterated ("derisked") variant of Qwen's Qwen3.8-Flash-Next. Multimodal (image-text-to-text), MTP speculative-decode head and PLE n-gram table preserved.

Built to run the 80B-class Flash-Next MoE on 2× AMD MI210 (gfx90a) at long context.

What this is

• —Format: compressed-tensors, W4A16 — 4-bit weights, group size 128, symmetric, no zero-point (loads on vLLM's TritonW4A16 + fused_moe_kernel_gptq_awq fast paths).
• —Method: AWQ (activation-aware). Smoothing is restricted to the mathematically-exact up_proj → down_proj pairs. The standard norm → q/k/v and norm → gate/up smoothings are deliberately excluded: this architecture uses hyper-connections with a data-dependent (sigmoid) gate, so folding a smoothing scale into the norm would change the gate non-linearly and corrupt the weights.
• —Quantized: routed experts (gate/up/down) and full-attention q/k/v/o.
• —Kept in bf16 (matching the reference recipe): the GDN linear-attention layers (recurrent state — error compounds along the sequence), the shared expert, the QSA indexer, the router gates, hyper-connections, norms, embed_tokens, lm_head, the MTP speculative-decode head, and the PLE n-gram embedding table.

Performance (measured, 2× MI210, vLLM, TP=2, 256K context)

Decode is bandwidth-bound: the ~100 GB PLE table is CPU-offloaded and streamed over PCIe, so single-token decode is inherently slow; the MTP head (spec decode) recovers most of it. Prefill is compute-bound and rides the tuned int4 MoE kernels.

Note on `num_speculative_tokens`: 3 is the practical ceiling on this stack — n≥5 trips a QSA-indexer constraint (ring capacity 12 must divide the attention block size).

Serving (vLLM, ROCm / MI210)

vllm serve Qwen3.8-Flash-Next-DERISKED-W4A16-AWQ \
  --tensor-parallel-size 2 --enable-expert-parallel --dtype bfloat16 \
  --gpu-memory-utilization 0.95 --max-model-len 262144 --max-num-seqs 4 \
  --block-size 64 --cpu-offload-params ngram_embedding --cpu-offload-gb 60 \
  --moe-backend triton --mamba-cache-dtype bfloat16 \
  --speculative-config '{"method":"qwen4_exp_mtp","model":"Qwen3.8-Flash-Next-DERISKED-W4A16-AWQ","num_speculative_tokens":3}' \
  --disable-custom-all-reduce --trust-remote-code

Provenance & credits

• —Base model: Qwen/Qwen3.8-Flash-Next — © Qwen.
• —Abliteration: Blackfrost-AI/Qwen3.8-Flash-Next-DERISKED-BF16.
• —Quantization: this repo (W4A16 / AWQ via llm-compressor).

License

Qwen Community License 1.0 (qwen-community-1.0), inherited from the base model. See the included LICENSE and the upstream Qwen3.8-Flash-Next LICENSE. This quantization is a derivative work and is distributed under the same terms.

Disclaimer

This is an abliterated / uncensored model: safety refusal behavior has been removed. You are responsible for how you use it and for compliance with the base-model license and applicable law.