codegyver/Qwen3.8-Flash-Next-NVFP4-FP8-PBWO-DGX-Spark

Qwen3.8-Flash-Next NVFP4 + FP8 PBWO — DGX Spark

A mixed-precision derivative of `Inferact/Qwen3.8-Flash-Next-NVFP4` built and validated for single-NVIDIA-DGX-Spark / GB10 inference with vLLM.

The routed MoE experts remain in the source checkpoint's NVFP4 format. The dense projections that execute on every token are converted to 128×128 blockwise FP8 weight-only (`FP8_PB_WO`):

• —12 QSA layers: q_proj, k_proj, v_proj, o_proj — 48 matrices
• —36 Gated DeltaNet layers: in_proj_qkv, in_proj_z, out_proj — 108 matrices
• —156 FP8 matrices total

Everything else is left unchanged from the Inferact source checkpoint, including the routed-expert payloads, shared experts, MTP tensor payloads, PLE, vision stack, embeddings, LM head, routers, norms, hyperconnections, and QSA indexer weights.

[!IMPORTANT] This is an experimental community derivative, not an official Qwen, Inferact, NVIDIA, or vLLM release. Throughput and serving behavior have been validated on one DGX Spark. A broad task-quality benchmark establishing equivalence to the source checkpoint has not yet been published; evaluate the model on workloads that matter to you before production use.

Why this variant exists

On a DGX Spark, leaving the non-expert QSA and Gated DeltaNet projections in their source precision means those dense weights are read on every generated token. The goal of this derivative is to reduce that recurring memory traffic while preserving the already-compact NVFP4 routed experts.

A QSA-only experiment was slower on the clean long-generation test. Converting both QSA + GDN changed the tradeoff and produced a measurable decode improvement on GB10.

This is therefore a deliberately targeted mixed-precision layout rather than a "quantize everything" checkpoint.

Precision layout

The conversion performs no additional training and uses no calibration dataset. FP8 weights are serialized for vLLM's native ModelOpt FP8_PB_WO path using FP8 E4M3 values with 128×128 weight blocks and the corresponding ModelOpt block-scale metadata.

The checkpoint also carries the runtime metadata aliases needed by the tested Qwen3.8 preview path, including the MTP runtime-layer mapping. The MTP weights themselves are not requantized.

DGX Spark performance

Development measurements were collected on one NVIDIA DGX Spark / GB10 using vLLM with MTP speculative decoding (num_speculative_tokens=3).

Clean long-generation comparison

The most directly comparable measurement is the fixed 2048-output-token LongCode case:

That is approximately +6.1% vs. the source checkpoint and +15.7% vs. the QSA-only experiment on this specific single-stream test.

Recorded development run

The shorter historical cases did not always produce identical completion lengths, so they should be treated as directional development measurements, not strict kernel-level A/B results. The LongCode case is the cleanest comparison because both runs generated exactly 2048 output tokens.

The exact prompt text from these archived development runs was not preserved. The companion GitHub repository includes a frozen public prompt set and benchmark harness for new reproducible comparisons.

Tested runtime

Runtime compatibility

This checkpoint uses a mixed ModelOpt quantization layout:

• —routed MoE experts remain NVFP4;
• —QSA and Gated DeltaNet projection weights use `FP8_PB_WO`;
• —MTP, shared experts, vision, PLE, embeddings, LM head, and the remaining tensor payloads are left unchanged.

The tested Qwen3.8 preview container:

vllm/vllm-openai:qwen38-flash-next

uses preview vLLM:

0.1.dev20073+g8e685d198

That build already contains vLLM's native:

ModelOptFp8PbWoLinearMethod

and it already understands Qwen3.8's packed/fused QKV naming. However, its ModelOptMixedPrecisionConfig predates the dispatcher branch that selects FP8_PB_WO for layers declared with:

{
  "quant_algo": "FP8_PB_WO"
}

Without that dispatch, vLLM creates those layers as unquantized even though the checkpoint contains FP8 PBWO weights and scale metadata. Loading then fails because the runtime layer does not have the parameters that the serialized FP8 checkpoint expects.

The runtime used for this release applies a small compatibility backport that connects:

ModelOpt MIXED_PRECISION
        +
quant_algo = FP8_PB_WO
        ↓
ModelOptFp8PbWoLinearMethod

No custom FP8 kernel is added, and this is not a maintained fork of vLLM. The patch simply enables an FP8 PBWO implementation that already exists in the preview image.

Current upstream vLLM already contains equivalent FP8_PB_WO support in the ModelOpt mixed-precision dispatcher, so this compatibility patch is specifically for the Qwen3.8 preview runtime used to validate this model.

The tested prebuilt runtime is available on Docker Hub:

aspotton/qwen38-flash-next-vllm:dgx-spark-fp8-pbwo-v1

This image is based on the official Qwen3.8 preview container and contains only the runtime compatibility change described above. It does not contain the model weights. The checkpoint is downloaded separately from Hugging Face.

DGX Spark / GB10 kernel workaround

The tested DGX Spark configuration also uses:

VLLM_USE_DEEP_GEMM=0

On GB10 (sm_121), the preview runtime can select the DeepGEMM block-FP8 path even though that path does not execute correctly in this configuration. Disabling DeepGEMM causes vLLM to use the working CUTLASS block-FP8 implementation instead.

This is why the documented launch command includes:

-e VLLM_USE_DEEP_GEMM=0

This setting is specific to the tested GB10 runtime path and should not be interpreted as a general requirement for FP8 PBWO on all hardware.

Why use the provided Docker image?

The prebuilt image exists to make the model immediately usable on DGX Spark while preserving reproducibility.

Instead of requiring users to:

1. 1.obtain the Qwen3.8 preview vLLM image;
2. 2.locate and patch its ModelOpt mixed-precision dispatcher;
3. 3.rebuild the container;
4. 4.verify the PBWO runtime mapping;
5. 5.configure the GB10 kernel workaround;

users can pull the known-good runtime directly:

docker pull \
  aspotton/qwen38-flash-next-vllm:dgx-spark-fp8-pbwo-v1

and serve the Hugging Face checkpoint with the same runtime used for the reported DGX Spark measurements.

The companion GitHub repository will publish the Dockerfile and compatibility patch as well, so the runtime can be inspected and rebuilt independently rather than treated as an opaque binary image.

Companion GitHub repository

A public GitHub repository containing the converter source, preview-vLLM compatibility Dockerfile, runtime checks, exact DGX Spark serving recipe, benchmark tooling, tests, and implementation notes is coming soon.

The GitHub repository will contain no model weights.

Host swap setup on DGX Spark

The tested DGX Spark host used a 128 GiB swap file. This provides additional host-memory headroom during model initialization and PLE CPU offload. It is not intended as a substitute for RAM during steady-state inference; sustained heavy swap activity will hurt performance.

Create the swap file using the exact size used for the tested setup:

# 137,438,953,472 bytes = 128 GiB
sudo fallocate -l 137438953472 /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Verify that it is active:

swapon --show
free -h

To make the swap file persistent across reboots, add it to /etc/fstab:

grep -qE '^/swapfile[[:space:]]' /etc/fstab || \
  echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

You can confirm the persistent entry with:

grep -E '^/swapfile[[:space:]]' /etc/fstab

After a reboot, verify again with:

swapon --show
free -h

[!NOTE] The model may run with a different swap configuration depending on available host memory and other workloads. The commands above document the configuration used for this DGX Spark setup and its reported measurements.

Quick start on DGX Spark

1. Runtime compatibility image

The runtime used for the results below is:

qwen38-flash-next:fp8-pbwo-mixed-v1

The Dockerfile and small vLLM preview compatibility backport used to build this image will be published in the companion GitHub repository soon.

2. Serve this model from Hugging Face

MODEL=codegyver/Qwen3.8-Flash-Next-NVFP4-FP8-PBWO-DGX-Spark

docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_PLE_CPU_OFFLOAD=1 \
  -e VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800 \
  -e VLLM_USE_DEEP_GEMM=0 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False \
  aspotton/qwen38-flash-next-vllm:dgx-spark-fp8-pbwo-v1 \
  "$MODEL" \
  --distributed-executor-backend mp \
  --served-model-name qwen \
  --max-num-seqs 8 \
  --kv-cache-memory=25770700084 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --enable-expert-parallel \
  --no-enable-flashinfer-autotune \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --block-size 32 \
  --performance-mode interactivity \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

The server exposes an OpenAI-compatible API on:

http://localhost:8000/v1

Local checkpoint

If the model is already present locally, use its mounted path instead of the Hub repository ID:

/models/qwen38-flash-next-inferact-nvfp4-fp8qsa-gdn-v3

That is the path used for the reported development run.

OpenAI-compatible request example

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen",
    "messages": [
      {
        "role": "user",
        "content": "Explain why reducing dense side-layer bandwidth can improve MoE decode speed."
      }
    ],
    "max_tokens": 512,
    "temperature": 0.2
  }'

Multimodal support

The source model's vision/multimodal weight path is not modified by this conversion. The derivative is intended to retain the source model's multimodal interfaces.

That said, unchanged weights are not the same thing as a complete quality certification. Run your own multimodal evaluations for production workloads. The companion GitHub repository includes a vision smoke test.

Conversion and reproducibility

This checkpoint is generated from:

`Inferact/Qwen3.8-Flash-Next-NVFP4`

using the public converter in the companion repository:

converter/convert_inferact_qwen38_fp8_pbwo_qsa_gdn_v3.py

The converter:

1. 1.discovers the 12 QSA and 36 GDN layers;
2. 2.converts exactly 156 dense projection matrices to blockwise FP8 E4M3;
3. 3.preserves the source NVFP4 routed experts;
4. 4.leaves all non-target tensor payloads unchanged;
5. 5.writes mixed ModelOpt quantization metadata;
6. 6.emits packed/fused runtime aliases required by the tested Qwen3.8 path;
7. 7.adds the MTP runtime-layer NVFP4 metadata alias;
8. 8.audits the output checkpoint before reporting success.

The source checkpoint is never modified by the converter.

Validation status

Validated so far:

• —checkpoint conversion and post-conversion audit;
• —model loading on one DGX Spark / GB10;
• —text generation through vLLM;
• —MTP speculative decoding with 3 speculative tokens;
• —OpenAI-compatible serving;
• —the throughput measurements documented above.

Not yet claimed:

• —broad benchmark parity with the source model;
• —statistically rigorous quality equivalence;
• —production behavior at high concurrency;
• —validation across multiple GPU architectures;
• —exhaustive multimodal quality parity.

Please report reproducible issues in the companion GitHub repository.

Prior art

This project does not claim that combining NVFP4 experts with FP8 side layers is a novel quantization concept. Closely related public work includes:

• —`lovedheart/Qwen3.8-Flash-Next-NVFP4-FP8` — NVFP4 routed experts with blockwise FP8 QSA/GDN, focused on a patched SGLang serving path.
• —`blazux/qwen3.8-Flash-DGX` — a DGX Spark serving project that also explores a hybrid FP8 side-layer layout.
• —`primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8` — a mixed NVFP4/FP8 vLLM checkpoint that quantizes QSA while keeping GDN unchanged in its published configuration.

The purpose of this release is narrower: provide a reproducible Inferact-derived, vLLM-oriented, QSA+GDN FP8_PB_WO checkpoint and tested DGX Spark runtime recipe, together with the converter and the measurements that led to the chosen allocation.

Source model and acknowledgements

This work builds on:

• —Qwen/Qwen3.8-Flash-Next
• —Inferact/Qwen3.8-Flash-Next-NVFP4
• —vLLM
• —NVIDIA TensorRT Model Optimizer

Thanks also to the authors of the related public projects listed above for their work on Qwen3.8 mixed-precision serving.

License

This derivative remains subject to the Qwen Community License 1.0 used by the source checkpoint. See the included `LICENSE` file for the complete terms.

The converter, Docker compatibility layer, benchmark tooling, and documentation in the companion GitHub repository are separately licensed there.

Disclaimer

This model is provided as-is. Quantization can change numerical behavior even when most of the checkpoint is left untouched. Validate accuracy, safety, tool use, multimodal behavior, and performance for your own deployment before relying on it.