CaptainArni/Swift-Qwen3.8-27B-NInfer

Swift-Qwen3.8-27B — NInfer v3 artifact

A single-file NInfer v3 artifact of ukisai/Swift-Qwen3.8-27B-NVFP4, UkisAI's reasoning-efficient fine-tune of Qwen3.8-27B in NVIDIA ModelOpt's NVFP4/FP8 quantization. Converted with upstream's own converter and the recipe included here; weight bytes are imported, not requantized.

What is inside: all 64 MLP layers and nothing else at NVFP4 (block size 16); the 144 GDN and 64 full-attention projections at FP8 E4M3 with one BF16 scale per row; an FP8 output head and FP8 embeddings; Q4/Q5/Q6 Vision; Q8 MTP; z-lab's DFlash2 draft at W8G32. Norms, convolutions, the small GDN projections and the draft codebooks stay BF16.

Quick start

Build NInfer (Linux + NVIDIA, CUDA 12.8+; tested on an RTX 5090) and serve the file directly — a v3 artifact carries its own config, tokenizer, chat template and bindings, so there is nothing else to download:

git clone https://github.com/Neroued/ninfer && cd ninfer && ./build.sh
hf download CaptainArni/Swift-Qwen3.8-27B-NInfer \
  qwen3_8_27b_nvfp4swift.ninfer --local-dir ~/ninfer-models

./build/apps/ninfer-serve ~/ninfer-models/qwen3_8_27b_nvfp4swift.ninfer \
  --host 127.0.0.1 --port 8080 \
  --max-context 64000 --kv-capacity 64000 --kv-dtype int8 \
  --max-concurrency 2 \
  --spec mtp --draft-tokens 3 --lm-head-draft

It serves an OpenAI-compatible API:

curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8-27b","messages":[{"role":"user","content":"What is 17*23?"}]}'

Speculative decoding — pick one at startup. --spec mtp --draft-tokens 3 uses Swift's own MTP head. --spec dflash2 --draft-tokens 7 uses the DFlash2 draft, which is faster on code and structured output and costs ~1.6 GiB more weights plus ~0.9 GiB of runtime. Only the selected backend is materialized.

Context and VRAM, measured on a 32 GiB card with Vision off:

Add --vision for images and video (~0.25 GiB of weights plus a media arena). The architecture ceiling is 262,144 tokens; --kv-dtype nvfp4 roughly halves the pool. The model id is qwen3.8-27b because the artifact was named to drop into a setup already serving that id — the weights are Swift's.

Measured against the official NVFP4 artifact

Against `neroued/Qwen3.8-27B-nvfp4-NInfer` (base Qwen3.8 in unsloth's NVFP4/FP8), RTX 5090, 224,000 context, Vision off, KV int8, MTP, one request at a time, both runs back to back. This compares two artifacts: the fine-tune and a different quantizer at once. Tokens and quality belong to the fine-tune; the speed mostly belongs to this file quantizing all 64 MLP layers where the official one leaves the last eight FP8.

Tokens — 14 tasks × 4 seeds, 32,768-token cap, temp 1.0 / topp 0.95 / topk 20, thinking at the template default (xhigh):

Fewer tokens on 14 of 14 tasks (one-sided sign test p = 0.00006), per-task ratio 0.57 geometric mean. UkisAI claim 41% fewer thinking tokens at xhigh; this measured 50% on a smaller, easier set. At tighter caps (4k/6k) the same prompts finished 17/18 against 8/18.

Speed and footprint:

What this does not show. The quality suite is a smoke test — six graded task types, two to four seeds — able to catch damage, not to resolve a point on a public eval. Decode figures come from one greedy pass per topic. In a 12-run multi-turn agent loop this artifact failed to end the turn twice under one agent where the official artifact did not, and passed 12/12 under another; at that sample size the difference is not significant (Fisher p = 0.22), but it is the open question on this file. The DFlash2 draft was trained against base Qwen3.8's hidden states, not Swift's, and still accepted 78% on a code prompt.

How it was built

swift_nvfp4.py in this repo is the recipe. Upstream reads compressed-tensors; this checkpoint is ModelOpt, which stores the same NVFP4 bits under other names and inverts two scales, so the recipe adds a reader for it. Conversion took 4 min 48 s on CPU and imports every NVFP4 and FP8 code word unchanged.

python3 -m tools.convert \
  --model /path/to/Swift-Qwen3.8-27B-NVFP4 \
  --recipe swift_nvfp4.py \
  --source dflash2=/path/to/Qwen3.8-27B-DFlash2 \
  --components text,vision,mtp,dflash2 \
  --resource chat_template.jinja=tools/chat_templates/qwen3_8.jinja \
  --name qwen3.8-27b --device cpu \
  --out qwen3_8_27b_nvfp4swift.ninfer

Two things differ from upstream's qwen3_8_27b_nvfp4 recipe, both forced: every MLP layer is NVFP4 (upstream stops at 56 because unsloth's checkpoint leaves the rest FP8), and the output head is re-encoded from NVFP4 to FP8 because linear_topk takes a Q8 or an FP8 head only. ModelOpt's per-tensor FP8 scale is carried as a BF16 per-row multiplier: measured against Swift's own matrices that is 0.06–0.30% relative error, where re-encoding each row from decoded values costs 2.4–2.8%.

Sources: Swift NVFP4 at revision 4cf10191, z-lab/Qwen3.8-27B-DFlash2 at 50307d4c, NInfer at 6cc95cc5.

License and credits

The weights are UkisAI's and this artifact inherits the Swift Open License v1.0 from them: free for personal, research, educational, evaluation and commercial use for individuals and organizations with annual recurring revenue up to US$1,000,000; above that, commercial use needs a Swift Enterprise License. See the original model for the terms and for access.

Thanks to UkisAI for Swift, to Neroued for NInfer and its converter, and to z-lab for the DFlash2 drafter.