d0xin/Swift-Qwen3.8-27B-FP8

Swift-Qwen3.8-27B-FP8

FP8 quantization of `ukisai/Swift-Qwen3.8-27b`.

This is an independent community quantization and is not an official UkisAI release.

The goal of this checkpoint is to preserve the behavior of Swift-Qwen3.8-27B while reducing VRAM requirements and enabling high-throughput inference with SGLang, including speculative decoding with the model's native MTP head or an external DFlash2 draft model.

Model summary

• —Upstream model: ukisai/Swift-Qwen3.8-27b
• —Architecture: Qwen3_5ForConditionalGeneration
• —Quantization format: compressed-tensors
• —Quantization scheme: FP8_BLOCK
• —Weight block size: 128 x 128
• —Activations: dynamic FP8
• —Activation group size: 128
• —Quantizer: llmcompressor 0.13.0
• —Declared context length: 262,144
• —Checkpoint size: approximately 29 GB
• —Native MTP components: retained

The checkpoint was produced from the BF16 Swift-Qwen3.8-27B model rather than requantizing an already quantized derivative.

Quantization details

The quantization process used the official Qwen3.8 FP8 configuration as a reference for the block-FP8 layout.

Checkpoint audit:

Matrices that are not compatible with the required 128 x 128 block structure were preserved instead of being forcibly quantized.

The following classes of tensors were intentionally preserved where appropriate:

• —embeddings
• —lm_head
• —normalization parameters
• —non-2D weights
• —matrices whose dimensions are incompatible with the FP8 block layout

The native MTP layers are retained. Compatible MTP projection matrices are quantized to FP8, while incompatible components remain unquantized.

Validation

Validated locally on:

• —NVIDIA RTX PRO 6000 Blackwell 96 GB
• —SGLang 0.5.19.dev135+ga4ffb996d
• —compressed-tensors 0.18.0
• —CUDA-capable Linux deployment
• —single-GPU tensor parallelism (TP=1)

SGLang successfully loads the checkpoint as:

type=Qwen3_5ForConditionalGeneration
quant=compressed-tensors

Observed target-model weight memory during loading:

28.47 GB

OpenAI-compatible /v1/chat/completions inference was validated successfully.

Multimodal inference has not yet been separately benchmarked for this quantized checkpoint.

Performance

All measurements below are local measurements from a single NVIDIA RTX PRO 6000 Blackwell 96 GB GPU.

They are intended to document this deployment, not to serve as standardized cross-model benchmarks.

Fixed 4,096-token generation

Same prompt and generation setup for all configurations:

Measured DFlash2 runs:

131.24 tok/s
132.65 tok/s
132.62 tok/s
median: 132.62 tok/s

Compared with target-only generation, DFlash2 produced approximately 2.63x higher output throughput in this test.

Compared with native NEXTN/MTP, DFlash2 was approximately 28% faster.

The DFlash2 draft model is external and is not included in this repository.

Reasoning-heavy agent test

A separate local A/B test used the same diagnostic prompt, sampling parameters, seed, and reasoning setting for both systems.

The prompt asked the model to diagnose an intermittently slow OpenAI-compatible inference deployment with high GPU utilization, large KV cache, speculative decoding, variable context sizes and concurrency-sensitive latency.

Observed change:

• —wall-clock time: approximately -27.8%
• —reasoning tokens: approximately -27.8%
• —completion tokens: approximately -29.2%

The main benefit in this test was not higher raw per-token throughput. Swift reached a similarly useful diagnostic answer with substantially fewer reasoning and completion tokens.

This is a local workload test and should not be interpreted as a standardized quality benchmark.

SGLang usage

Basic serving

python -m sglang.launch_server \
  --model-path /path/to/Swift-Qwen3.8-27B-FP8 \
  --served-model-name Swift-Qwen3.8-27B-FP8 \
  --host 0.0.0.0 \
  --port 30000 \
  --attention-backend flashinfer \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

Native NEXTN / MTP speculative decoding

The retained native MTP head can be used with SGLang:

python -m sglang.launch_server \
  --model-path /path/to/Swift-Qwen3.8-27B-FP8 \
  --served-model-name Swift-Qwen3.8-27B-FP8 \
  --host 0.0.0.0 \
  --port 30000 \
  --attention-backend flashinfer \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

In the validated SGLang build, NEXTN is internally represented through the EAGLE speculative-decoding path.

DFlash2 speculative decoding

Best local throughput was obtained with a compatible external DFlash2 draft checkpoint:

python -m sglang.launch_server \
  --model-path /path/to/Swift-Qwen3.8-27B-FP8 \
  --served-model-name Swift-Qwen3.8-27B-FP8 \
  --host 0.0.0.0 \
  --port 30000 \
  --attention-backend flashinfer \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path /path/to/Qwen3.8-27B-DFlash2 \
  --speculative-num-draft-tokens 8

The DFlash2 weights are not redistributed in this repository.

Because speculative decoding verifies proposed tokens against the target model, the external draft model affects acceptance rate and speed rather than replacing the target model's token distribution.

Notes

This repository contains the quantized target checkpoint only.

It does not include:

• —a DFlash2 draft checkpoint
• —the original BF16 Swift checkpoint
• —SGLang runtime binaries or containers

Performance depends heavily on GPU architecture, SGLang version, attention backend, context length, concurrency, KV-cache configuration and speculative decoding parameters.

License

This checkpoint is derived from:

`ukisai/Swift-Qwen3.8-27b`

and follows the Swift Open License v1.0 applicable to the upstream model.

Please refer to the upstream repository and its license text for the authoritative licensing terms.

No additional rights to the upstream model are granted by this repository.

Attribution

Original model:

• —UkisAI
• —ukisai/Swift-Qwen3.8-27b

FP8 conversion, validation and local performance measurements for this repository were performed independently by the repository maintainer.

Citation

For the underlying Swift model, please cite or reference the upstream project:

@misc{swift-qwen3.8-27b,
  title  = {Swift-Qwen3.8-27B},
  author = {UkisAI},
  year   = {2026},
  url    = {https://huggingface.co/ukisai/Swift-Qwen3.8-27b}
}