Dragoy/Swift-Qwen3.8-27B-abliterated-NVFP4-DFlash2

Swift-Qwen3.8-27B · huihui-style abliterated · NVFP4 + DFlash 2 (built-in draft)

NVFP4 + FP8 quantized checkpoint of the abliterated Swift model, published in standard Hugging Face safetensors format, with the DFlash 2 draft model bundled in this repository. The drafter lives in `dflash2/` and config.json carries the matching speculative_config block (relative dflash2 path): engines that auto-read that key activate the drafter out of the box, while vLLM currently needs the explicit flag from the Usage section.

The target weights, tokenizer and behaviour are the standard-HF NVFP4 checkpoint — the full provenance chain (abliteration → quantization → config repair) is documented in the base repository.

Layout

DFlash 2 draft

• —Source: z-lab/Qwen3.8-27B-DFlash2 @ rev 50307d4c4cde6860d4eee73e2547cd786fe8e8a4 (Apache-2.0)
• —Block-diffusion drafter: 1.92B params, BF16 (hence 3.85 GB), 5 layers, sliding window 2048, block size 8; conditions on target layers [5, 19, 33, 47, 61]
• —dflash2/model.safetensors sha256 67fc76d68dc5a9415511a4f394ef744d67510cd20e93b37cc2cc7d28e4bab65c
• —dflash2/config.json sha256 873e3556509b0da06e29654ba00d4944888d4b5e8a33afde25f7eb27d321e980

The draft proposes up to 7 tokens per step; the target verifies the whole block in one forward pass, so the sampling distribution is exactly the target's — lossless.

Usage (vLLM)

pip install "vllm>=0.29"        # dflash2 drafter needs >= 0.29 (0.27.1 has no DFlash2DraftModel)
export VLLM_USE_FLASHINFER_SAMPLER=0
huggingface-cli download dragoy/Swift-Qwen3.8-27B-abliterated-NVFP4-DFlash2   --local-dir ./model          # target + dflash2/ drafter in one folder

vllm serve ./model   --max-model-len 32768 --gpu-memory-utilization 0.9   --speculative-config '{"method": "dflash", "model": "./model/dflash2", "num_speculative_tokens": 7}'

config.json also carries the same speculative_config block (with the relative dflash2 path) so that engines which auto-load it activate the drafter out of the box; vLLM 0.29.0 currently resolves model against the working directory / HF repo ID, hence the explicit flag above.

SGLang:

python -m sglang.launch_server   --model-path ./model   --speculative-algorithm DFLASH   --speculative-draft-model-path ./model/dflash2   --speculative-num-draft-tokens 8

Verified on target

Measured on one RTX PRO 6000 (sm120a), vLLM 0.27.1, 8 concurrent prompts × 512 max tokens, temp 1.0 / topp 0.95 / top_k 20, thinking xhigh; draft acceptance rate n/a. Throughput is inference-only (engine startup excluded). Greedy/sampled output is lossless vs the target.

Provenance

1. 1.Source: ukisai/Swift-Qwen3.8-27b @ 1b30aaaf753fe5c1cb51ada2ea0367a53445359c
2. 2.Abliteration: huihui-style (refusal-direction projection, Arditi et al. 2024 family), transferred by per-tensor weight difference from the Qwen/Qwen3.8-27B ↔ huihui-ai/Huihui-Qwen3.8-27B-abliterated pair — format/weight change, no fine-tuning
3. 3.Quantization: llm-compressor oneshot, NVFP4 + FP8 allocation copied verbatim from unsloth/Qwen3.8-27B-NVFP4, 32 calibration samples
4. 4.Config repair: nested multimodal config restored + quantization_config grafted (llm-compressor flattens it)
5. 5.DFlash 2 draft: bundled from z-lab/Qwen3.8-27B-DFlash2 @ 50307d4c (byte-identical, sha256 above); speculative_config added to config.json
6. 6.Verification: vLLM 0.29.0 smoke (baseline vs dflash) on RTX PRO 6000, Modal

License

The target is a derivative of the Swift checkpoint — Swift Open License v1.0 (`LICENSE`), not Apache. The bundled DFlash 2 draft is Apache-2.0 (z-lab); the Qwen base model underneath is also Apache-2.0. Practical terms of the Swift Open License (free use, commercial included, under $1,000,000 FY gross-revenue threshold; see the base repository for the full breakdown):

This is a description of what the licences say, not legal advice.