bjonor/Swift-1.5-Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16

Swift 1.5 Qwen3.8-27B — GPTQ INT4 (W4A16, group-128, symmetric) + BF16 MTP head

Unofficial 4-bit GPTQ quantization of `ukisai/Swift-1.5-Qwen3.8-27b` for low-VRAM inference, with the MTP (multi-token-prediction) draft head and the vision tower deliberately kept unquantized. Produced with gptqmodel 7.3.2 on an Intel Arc Pro B70 (30.3 GiB VRAM), ~2.4 h wall time. Quantization contract is identical to the Swift 1.0 artifact `bjonor/Swift-Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16`.

This model is not affiliated with, endorsed by, or supported by UkisAI or Alibaba Cloud. It is a community quantization; the licence terms of the base models below continue to apply (see License).

Runtime status (2026-09-25). Verified on the stock vllm/vllm-openai-xpu:nightly (0.29.1rc1.dev422+gd05da62e9.xpu, vllm-xpu-kernels 0.1.14.1) with the two runtime patches from the recipe repo: ready in 131 s, MTP-3 mean acceptance length 3.04 (68.0%), solo decode 61.0 tok/s (p512/g128, median n=5), endpoint + streaming smoke test PASS. patch_mtp_boundary.py is still required for requests that end exactly at --max-model-len; mixed prefill + spec-decode batches run without the GDN split-dispatch backport on this stack.

Model details

The quantize_config.json is field-for-field identical to the community reference artifact `SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16` except for the quant-time-only meta.offload_to_disk flag. Note that the reference artifact quantizes the base Qwen3.8-27B; this repository quantizes the Swift 1.5 fine-tune.

What was changed vs the source checkpoint

• —All linear projections of the language model → GPTQ INT4 (W4A16, g128, sym, desc_act=false).
• —mtp.* draft tensors (15) and model.visual.* vision tensors (333) kept in their original dtype; the dynamic exclusion {"-:.*mtp.*": {}} in quantize_config.json records this.
• —Added: quantize_config.json, quant_log.csv (per-module quantization timings).
• —Unchanged: config.json, generation_config.json, chat_template.jinja, tokenizer.json, tokenizer_config.json, vocab.json, merges.txt, processor_config.json, video_preprocessor_config.json.
• —Added licence files: LICENSE (Swift Open License v1.0), LICENSE-APACHE-2.0, NOTICE.

Quantization details

How to use

vLLM (Intel XPU) — tested configuration

vllm/vllm-openai-xpu:nightly (0.29.1rc1.dev422+gd05da62e9.xpu, vllm-xpu-kernels 0.1.14.1), with patch_mtp_nightly.py and patch_mtp_boundary.py applied at container start (launch.sh does this automatically):

vllm serve <this-repo> \
  --quantization gptq --dtype float16 --max-model-len 131072 \
  --gpu-memory-utilization 0.94 --kv-cache-dtype fp8 \
  --max-num-seqs 2 --max-num-batched-tokens 8192 --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --language-model-only

Measured on this artifact: ready after 131 s, mean acceptance length 3.04 (avg draft acceptance 68.0%), solo decode 61.0 tok/s median (n=5), short-prompt TTFT ~0.2 s, endpoint and streaming smoke tests pass. The pinned vllm 0.27.2rc1.dev77+gac7509e2b.xpu / vllm-xpu-kernels 0.1.12.3 stack with the derived vllm-xpu-gdn-split:0.1.12.3-p1 image also works (--max-num-seqs 1).

Notes for this base model on XPU:

• —MTP draft must be built unquantized. The checkpoint flags this via the dynamic exclusion; the runtime patch (or B70_MTP_BF16_DRAFT=1) enforces it on XPU builds.
• —`patch_mtp_boundary.py`. With spec decoding enabled, an unpatched engine dies when a request runs to exactly --max-model-len (EngineDeadError: Expected spec_token == num_spec_decodes * (num_speculative_tokens + 1) — the final draft group is truncated).
• —Mixed-batch limitation (pinned stack only). vllm-xpu-kernels < 0.1.14.1 abort the engine when speculative-decode tokens and prefill tokens land in the same invocation. Use --max-num-seqs 1, kernels ≥ 0.1.14.1, or the Python split-dispatch backport used by the pinned derived image. Not needed on the nightly stack above.
• —--kv-cache-dtype fp8 is a serving choice, not part of the checkpoint.

Full reproduction path — pinned environments, quant_swift.py / verify_quant.py, launch.sh (MTP and vision flags), the XPU patches and the benchmark harness — is in the recipe repository.

Transformers

GPTQ checkpoints need a GPTQ-capable loader (gptqmodel or auto-gptq) and transformers ≥ 4.5x. The mtp.* tensors are not used by transformers inference and can be ignored.

Evaluation

What was actually measured on this artifact (host: Arc Pro B70, vLLM XPU nightly, MTP 3 speculative tokens, fp8 KV cache, --max-num-seqs 2):

No standard quality benchmarks (MMLU, GPQA, AIME, IFBench, perplexity) were run on this quantized artifact by the uploader, and the concurrency harness (12-request storm / mixed batches) was validated on the Swift 1.0 artifact of the same architecture and contract, not re-run here. UkisAI publishes INT4 W4A16 evaluations for the base model (GPQA-Diamond 88.38%, IFBench 71.25%, AIME 2026 84.00%) which are indicative but were measured on a different quantization pipeline. Treat accuracy figures as unverified for this artifact.

Intended use and out-of-scope

• —Intended: local inference and evaluation of the Swift 1.5 fine-tune on Intel XPU / low-VRAM setups, including agent-style tool calling via qwen3_xml.
• —Out of scope: any safety-critical, medical, legal, or production decision making; anything requiring verified accuracy on this specific checkpoint; vision/multimodal use — the vision tower is preserved but was not validated (serving was tested language-only), and training data provenance is inherited from the base models.
• —No safety alignment or red-teaming was performed by the uploader. Behavioural risks of the base models (bias, hallucinations, prompt-injection susceptibility) also apply here and may be amplified by quantization.

Limitations

• —Quantization is lossy; per-module error was not published beyond the RTN fallback threshold report.
• —The MTP draft head only works in runtimes that can build it unquantized (see above); otherwise disable speculative decoding.
• —Exporting to other formats (GGUF/AWQ) from this checkpoint is untested.
• —Multimodal inputs are structurally supported but untested.
• —Long-context behaviour was validated to 131,072 tokens, not to the native 262,144; the concurrency storm was not re-run on this artifact (see above).

Environmental impact

One quantization run: ~2.4 h on a single Arc Pro B70 (230 W board power cap) — roughly 0.55 kWh board energy, plus host overhead (i9-13900K, 64 GB RAM). Serving costs depend on runtime and context length.

License and attribution

The following notice is required by the Swift Open License v1.0 for redistributors of quantizations/conversions/merges (verbatim from the base model's LICENSE appendix):

Copyright 2026 UkisAI. Swift Contribution licensed under the Swift Open License v1.0 (https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-27b/blob/main/LICENSE). Derivative of Qwen3.8-27B, Copyright 2026 Alibaba Cloud, Apache License 2.0.

Included files: `LICENSE` (Swift Open License v1.0, governs the Swift Contribution), `LICENSE-APACHE-2.0` (governs Qwen3.8-27B and the files unmodified from it), `NOTICE` (what UkisAI changed).

Key terms to be aware of:

• —Commercial-use threshold. Personal, research, educational, evaluation and commercial use are free for individuals and organisations with gross annual revenue (including affiliates) up to US$1,000,000. Above that threshold, commercial use of the Swift Contribution requires a separate Swift Enterprise License from UkisAI. Nothing in the Swift Open License limits your rights in Qwen3.8-27B itself, which remains Apache-2.0.
• —No trademark rights. The licence grants no rights to the "UkisAI" name or marks; this card uses them only for attribution.
• —Termination. Non-compliance terminates the licence for the Swift Contribution; rights in the base model under Apache-2.0 are unaffected.

Addendum for this quantization (the uploader's modification notice):

GPTQ INT4 quantization of the Swift Contribution by Bjorn Nordblom, 2026, distributed under the same Swift Open License v1.0 terms that apply to the Swift Contribution. No additional rights are granted.

Citation

@misc{swift-1.5-qwen3.8-27b-gptq-int4,
  title        = {Swift 1.5 Qwen3.8-27B GPTQ INT4 (W4A16, group-128, symmetric) with preserved BF16 MTP head},
  author       = {Nordblom, Bjorn},
  year         = {2026},
  howpublished = {Hugging Face},
  note         = {Quantization of ukisai/Swift-1.5-Qwen3.8-27b},
}

@misc{swift-1.5-qwen3.8-27b,
  title  = {Swift 1.5 Qwen3.8-27B},
  author = {UkisAI},
  year   = {2026},
  url    = {https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-27b}
}

@misc{qwen3.8-27b,
  title  = {Qwen3.8-27B},
  author = {Alibaba Cloud},
  year   = {2026},
  url    = {https://huggingface.co/Qwen/Qwen3.8-27B}
}

Acknowledgements

• —UkisAI for the Swift 1.5 fine-tune and the licence terms above.
• —Alibaba Cloud / Qwen for Qwen3.8-27B (Apache-2.0).
• —The gptqmodel project for the quantizer.
• —SergiioB's intel-arc-pro-b70-inference-cookbook for the Intel Arc Pro B70 XPU serving patches (MTP BF16 draft, GDN boundary handling, mixed-batch split-dispatch backport); the copies vendored in the recipe repo are MIT, Copyright (c) 2026 SergiioB.

Model card contact

Issues and questions: https://github.com/BjornNordblom/intel-arc-b70-quant/issues