windowsxp811203/Qwen3.8-Flash-Next-Abliterated-NVFP4

Qwen3.8-Flash-Next-Abliterated — NVFP4 (W4A16)

NVFP4 build of windowsxp811203/Qwen3.8-Flash-Next-Abliterated, which is a refusal-removed Qwen3.8-Flash-Next. 336 GiB → 173.6 GiB.

This is weight-only NVFP4 (quant_algo: W4A16_NVFP4): activations stay in bf16, so there are no activation scales and no FP4 tensor-core compute is required. Whether a given runtime will serve it on pre-Blackwell hardware depends on whether that runtime has a dequantizing FP4 kernel path; do not read the NVFP4 name as a hardware requirement without checking your runtime.

Read this before using it. The weights are built and structurally verified tensor by tensor, but this checkpoint has not been served end to end here, so no quality benchmark was run on the quantized weights — the numbers further down are the bf16 parent's. Two reasons, and they are different: no released inference server supports qwen4_exp yet (vLLM PR #53896 and SGLang PR #36497 are both open, and people are demonstrably running those branches), and the host these weights were built on could not run either branch — it has no container runtime and CUDA 12.6, while sm_120 kernels need ≥12.8. If you serve this and measure it, please open a discussion; when it is measured here this card gets numbers or a retraction.

What is quantized, and what the size actually buys

The MTP head is 4.9 GiB rather than a rounding error because it carries its own fused 512-expert MoE (mtp.layers.0.mlp.experts.{gate_up_proj, down_proj} account for 4.7 of the 4.9), left in bf16 along with the rest of the head. Regenerate this table with nvfp4_breakdown.py.

Only the experts are quantized — but they are 120.8 B of the model's 180.0 B parameters, so that single family is where essentially all of the compression comes from. The n-gram table is the reason this file is 173.6 GiB rather than ~70: it is a 51.2 B-parameter lookup table, and it is left alone on purpose (see below).

Three decisions worth explaining

1. W4A16, not W4A4. The most-downloaded NVFP4 build of the base model (RadixArk, which its own card describes as a private candidate release) is W4A4: its config.json enables 4-bit input activations with dynamic: false, i.e. scales fixed at calibration time rather than computed per batch. Abliteration is defined as moving the activation distribution — it orthogonalizes every residual writer against a refusal direction. So those scales cannot be reused, and freshly calibrated ones would carry a quality cost that nobody has measured on an abliterated model. Weight-only has no activation scales at all, so the question does not arise. quant_algo in config.json reads W4A16_NVFP4.

2. The n-gram table stays bf16. Quantizing it to FP8 saves ~47 GB and is what RadixArk's build does (its shard headers show 51.2 GB of F8_E4M3). Doing that makes correctness depend on the runtime taking an FP8-specific code path for the PLE embedding rather than just upcasting the bytes. I have not demonstrated a loader getting that wrong — no runtime supports this architecture yet, so the concern is a reasoned one, not a measured one. With nothing able to serve it either way, the variant that removes the question seemed worth 47 GB.

Community variant (FP8 PLE). gorbatjovy built exactly that FP8-PLE version — qwen3.8-flash-next-abliterated-NVFP4-plefp8 — repacking the n-gram table to F8_E4M3 for 173.6 → 125.95 GiB (135.2 GB), experts / MTP head / vision tower untouched. Metadata-verified here (only the PLE dtype changes), and he measured it against this model — top-1 token agreement 100 %, mean KL 0.0018, median 0.0006, p90 0.0060 — i.e. effectively lossless, which is what you'd expect from FP8 on a pure lookup table with no compute path for error to accumulate through. That is real evidence the loader-path concern above does not bite in at least one runtime. Full credit to him; if you want the smaller file and your loader handles FP8 embeddings, that's the one to grab.

3. The vision tower and MTP head were grafted back — both halves. Other published NVFP4 builds of this architecture do retain both, so this is not a differentiator; it is a trap that had to be avoided, and it is worth being explicit about because it is silent when you fall into it. Quantizing through AutoModelForCausalLM never instantiates model.visual.* or mtp.*; export_hf_checkpoint then writes a checkpoint that simply does not contain them. The result loads, serves text, and looks complete — it has just stopped being multimodal and lost its draft head. Verified here:

  vision tower      333 tensors, all bf16/f32  [OK]
  MTP head           31 tensors, all bf16/f32  [OK]
  config.json (shipped)  ignore=294 entries | visual=True mtp=True

The 294-entry ignore list lives in the shipped config.json's embedded quantization_config — that is the authoritative source and what a loader reads. modelopt also drops an hf_quant_config.json build intermediate, but it carries an empty ignore list, so it is deliberately not shipped: publishing it would tell a compressed-tensors loader to quantize the grafted bf16 vision + MTP tensors — the exact silent failure this graft exists to prevent.

There is a third half, which the first version of this build got wrong. modelopt exports whatever class it was handed, so quantizing through AutoModelForCausalLM also emitted a text-only config.json — architectures: ["Qwen4ExpForCausalLM"], model_type: "qwen4_exp_text", no vision_config. The 333 vision tensors were in the shards and the config declared a model with nowhere to put them: present, and unreachable. fix_nvfp4_config.py (in this repo) rewraps modelopt's flat text config into the parent's multimodal shape and restores the preprocessor configs; the shipped config.json now loads as Qwen4ExpConfig / qwen4_exp with vision_config present. The text-only original is kept alongside as config.json.text-only so the difference is inspectable.

The ignore entries are the other half. Without them a modelopt/compressed-tensors loader treats the grafted bf16 tensors as quantization targets, finds no scales, and either fails outright or — in the MTP case — rejects every draft while the logs stay perfectly clean. I documented that failure mode for Qwen3.8-27B in nvfp4-mtp-survey; on this architecture it takes the vision tower with it.

Verification

Structural, on the shipped files, with verify_nvfp4.py (in this repo):

tensors: 222619  shards: 10  size: 173.6 GiB
index consistency: 0 declared-but-absent, 0 present-but-undeclared
experts: 73728 weight tensors (expect 73728), 73728 low-bit,
         73728 block scales, 73728 global scales
         dtypes: weight=U8 scale=F8_E4M3 scale_2=F32
NVFP4 EXPORT VERIFIED

73728 = 48 layers × 512 experts × 3 projections. That count is the whole point: on this architecture the experts are fused nn.Parameters of shape (512, out, in), not Linear modules, so a recipe that selects Linear by walking named_modules() skips 67 % of the model and still emits a well-formed quantized-looking config. nvidia-modelopt 0.46.0 detects the fused case explicitly (Detected fused MoE experts ... registering with _QuantFusedExperts) and un-fuses it on export, which is why the output namespace is layers.N.mlp.experts.E.{gate,up,down}_proj rather than the source's fused mlp.experts.down_proj.

Count tensors in your output. Do not read `quantization_config` and assume.

Parent model quality (bf16, not measured on these quantized weights)

Full method, the λ sweep, the paired McNemar test on MMLU and the byte-level verification are in the parent model card. The architecture write-up that all of this rests on is ARCHITECTURE.md.

Running it

No released inference server supports qwen4_exp yet — transformers 5.16.1 does, but it cannot load a modelopt-quantized checkpoint. When vLLM PR #53896 merges, the usual form applies:

vllm serve windowsxp811203/Qwen3.8-Flash-Next-Abliterated-NVFP4 --max-model-len 32768

Until then this repo is weights plus a reproducible recipe, not a deployable model. Reproduce with quantize_nvfp4_flashnext.py → graft_nvfp4.py → fix_nvfp4_config.py → verify_nvfp4.py, all included, plus nvfp4_breakdown.py for the size table above.

License

Qwen Community License 1.0 — not Apache-2.0 — inherited from the base model. Publishing and redistributing derivative weights is permitted. Running a Model-as-a-Service or AI Work Assistant business requires a separate license from Qwen before any commercial use; internal use is exempt provided the model, its outputs and its capabilities reach no third party.

Support / 打賞

If these models are useful to you, tips are appreciated — they pay for the GPU time. 如果這些模型對你有幫助，歡迎打賞，用於支應算力成本。

USDT (TRC20) · TPTo32r7vKazpTNaFqfFZ2ztoK1DG88888

Disclaimer

This model will not refuse. It is published for alignment and safety research. You are responsible for your use of it and for complying with applicable law.