tlwu/Qwen3.8-27B-NVFP4-ONNX

Qwen3.8-27B-NVFP4-ONNX

CUDA ONNX Runtime GenAI export of unsloth/Qwen3.8-27B-NVFP4, including the model's dense multi-token prediction (MTP) head for speculative decoding. The target decoder and MTP graph are packaged in one directory; no separate draft-model download is required. The 48 recurrent layers use the fused GatedDeltaNet operator and a four-slot state window so partially accepted MTP verification rounds can commit without a target-model replay.

This is a text-only ONNX export. The upstream Qwen3.8 checkpoint is multimodal, but this repository does not contain a vision encoder or image/video preprocessing graph.

Model details

The checkpoint's compressed weights are preserved, but the current H200 execution path is W4A16 for NVFP4 and W8A16 for FP8: weights remain quantized while activations are FP16. The eight dense MTP matrices are FP16. This all-FP16 MTP configuration adds 379.688 MiB over the block-32 INT8 MTP variant and was selected because it improved end-to-end decode throughput in the validation workloads where the additional memory was acceptable.

The package is approximately 21.1 GiB. Its largest files are:

Compatibility

This export was validated on an NVIDIA H200 (SM90) with CUDA 13.0, ONNX Runtime 1.30.0 development sources, and ONNX Runtime GenAI 0.16.0-dev. It uses recently added operators and MTP runtime APIs, so released packages available at the time of writing are not sufficient.

Use:

1. 1.ONNX Runtime built from `tlwu/20260819/gdn`. The exact validated commit is 044d929857cd5c0411f6f95b2495a249f00af7c5.
2. 2.ONNX Runtime GenAI built from `tlwu/qwen_gdn`. The exact validated commit is 03e9b4587c7c2212990276b5b2f7693beccbc887.
3. 3.An NVIDIA GPU with enough memory. Short-context single-stream validation used roughly 29-30 GiB, so a nominal 32 GiB GPU does not provide comfortable operational headroom.

The validated GenAI commit keeps greedy MTP verification inputs on device and uses one shared recurrent-state buffer under CUDA graph capture. Leave ORTGENAI_SHARE_RECURRENT_STATE_UNDER_GRAPH_CAPTURE unset for the recommended mode; setting it to 0 opts back into double buffering for diagnosis.

Other CUDA architectures have not been validated for this upload. Performance, supported quantized kernels, and memory use can differ by GPU and CUDA version.

Download

Install the Hugging Face CLI and download the complete repository:

python -m pip install -U "huggingface_hub[cli]"
hf download tlwu/Qwen3.8-27B-NVFP4-ONNX \
  --local-dir ./Qwen3.8-27B-NVFP4-ONNX
cd Qwen3.8-27B-NVFP4-ONNX
sha256sum -c SHA256SUMS

Do not download only one ONNX file. Both graphs, both external-data files, the tokenizer, and genai_config.json are required.

Build ONNX Runtime and GenAI

The commands below create a Python environment and build both required CUDA wheels from source on Linux. The small ORT_HOME directory gives GenAI the headers and shared libraries from the matching custom ONNX Runtime build.

Prerequisites include Python 3.10 or newer, Git, CMake 3.26 or newer, a C++20 compiler, and an installed CUDA toolkit. Set CUDA_HOME if the toolkit is not in its standard location.

python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
export CUDA_HOME=${CUDA_HOME:-/usr/local/cuda}
export CUDNN_HOME=${CUDNN_HOME:-$CUDA_HOME}

git clone --branch tlwu/20260819/gdn --single-branch \
  https://github.com/microsoft/onnxruntime.git
cd onnxruntime
./build.sh --config Release --parallel --use_cuda \
  --cuda_home "$CUDA_HOME" --cudnn_home "$CUDNN_HOME" \
  --build_shared_lib --build_wheel --skip_tests
python -m pip install --force-reinstall \
  build/Linux/Release/dist/onnxruntime_gpu-*.whl

cd ..
export ORT_SRC=$PWD/onnxruntime
export ORT_BUILD=$ORT_SRC/build/Linux/Release
export ORT_HOME=$PWD/ort-home
mkdir -p "$ORT_HOME/include" "$ORT_HOME/lib"
cp "$ORT_SRC"/include/onnxruntime/core/session/*.h "$ORT_HOME/include/"
cp "$ORT_SRC"/include/onnxruntime/core/session/*.inc "$ORT_HOME/include/" 2>/dev/null || true
cp "$ORT_SRC"/include/onnxruntime/core/providers/cpu/cpu_provider_factory.h \
  "$ORT_HOME/include/"
cp -P "$ORT_BUILD"/libonnxruntime.so* "$ORT_HOME/lib/"
cp "$ORT_BUILD"/libonnxruntime_providers_cuda.so \
  "$ORT_BUILD"/libonnxruntime_providers_shared.so "$ORT_HOME/lib/"

git clone --branch tlwu/qwen_gdn --single-branch \
  https://github.com/microsoft/onnxruntime-genai.git
cd onnxruntime-genai

python -m pip install -r requirements-dev.txt
python build.py --use_cuda --cuda_home "$CUDA_HOME" --ort_home "$ORT_HOME" \
  --config Release --parallel --skip_tests --skip_examples
python -m pip install --no-deps --force-reinstall \
  build/Linux/Release/wheel/onnxruntime_genai_cuda-*.whl

For a strictly reproducible build, check out the exact commits listed in the compatibility section after cloning. Do not substitute a generic ONNX Runtime nightly: it may not contain the GatedDeltaNet schema and CUDA kernel required by text.onnx.

Verify the installed packages:

python -c "import onnxruntime as ort; import onnxruntime_genai as og; print(ort.__version__, og.__version__)"

Python example: one-directory MTP generation

Although all files live in one directory, og.MtpGenerator uses two og.Model sessions: one for text.onnx and one for mtp.onnx. The maintained `qwen-3.6-mtp.py` example supports this one-directory layout despite its historical filename. It constructs the second session with an in-memory model.mtp overlay and preserves the shared initializer manifest.

From the directory that contains both cloned repositories, run:

CUDA_VISIBLE_DEVICES=0 ORT_MTP_PREFILL_CHUNK=1024 \
python onnxruntime-genai/examples/python/qwen-3.6-mtp.py \
  --model_path ./Qwen3.8-27B-NVFP4-ONNX \
  --max_draft_tokens 3 \
  --max_length 262144 \
  --max_new_tokens 1024 \
  --prompts "Explain why the sky appears blue during the day."

The example intentionally uses greedy decoding so its acceptance and throughput statistics are reproducible. Its built-in og.MtpGenerator path is the recommended implementation. The --reference option runs an educational pure-Python draft/verify loop and is substantially slower.

Sampling recommendations

The packaged configuration defaults to deterministic greedy decoding. When sampling is enabled, Qwen recommends:

ONNX Runtime GenAI 0.16.0-dev does not support min_p or a native presence_penalty. Their recommended thinking-mode values are zero and therefore no-ops. The recommended instruct-mode presence_penalty=1.5 cannot currently be represented faithfully; repetition_penalty is not equivalent and should not be used as a substitute.

max_length is the total sequence length, including the prompt. This export supports 262,144 total tokens. For example:

max_length = min(262144, prompt_length + requested_new_tokens)

Separate reasoning and final-answer token budgets are not supported by this runtime. Both parts occupy the same generated sequence and must fit, together with the prompt, within the 262,144-token total limit.

Validation notes

Validation used batch size 1, CUDA graph execution, a 1,024-token MTP prefill chunk, and clean processes on an H200. The recommended setting is max_draft_tokens=3; the four-slot state window supports at most three draft tokens. Performance is prompt- and hardware-dependent.

For an exact 2,048-token synthetic prompt followed by 2,048 greedy tokens, N=3 achieved:

A graph-stable sampled pilot used five requests from each task, with each sampled CUDA graph trajectory warmed to hash convergence before timing:

These are preliminary throughput measurements, not quality scores. The five-request sample is too small for small cross-workload comparisons, and cold generators also pay one-time CUDA graph capture costs. A fully compressed MTP head did not win consistently, so this package retains the original eight FP16 dense matrices and native FP8 LM head.

Repository contents

• —text.onnx and text.onnx.data: GatedDeltaNet target text decoder with a four-slot recurrent-state window.
• —mtp.onnx and mtp.onnx.data: dense FP16 MTP draft head.
• —genai_config.json: target/MTP graph metadata, shared initializer manifests, CUDA graph option, and default thinking-mode sampling parameters.
• —config.json: source architecture metadata.
• —tokenizer.json, tokenizer_config.json, and chat_template.jinja: text tokenizer and prompt template assets.
• —SHA256SUMS: checksums for all required serving files.

Attribution and provenance

This is a derivative conversion; no training or fine-tuning was performed here.

1. 1.Alibaba's Qwen team created the Apache-2.0 Qwen3.8-27B base model.
2. 2.Unsloth produced the mixed-precision NVFP4/FP8 checkpoint used as the conversion source.
3. 3.This repository converts and packages the text decoder and dense MTP head for ONNX Runtime GenAI CUDA execution, with calibrated INT8 target KV cache and shared initializers.

Limitations

• —Text generation only; vision and video inputs are not supported by this export.
• —Validated on H200/SM90 only at the time of publication.
• —Requires development/nightly runtime components and a source-built GenAI branch.
• —state_window=4 limits MTP to at most three draft tokens per target verification.
• —The 262,144-token limit includes both prompt and output.
• —Sampling is nondeterministic unless a fixed random_seed is supplied.
• —Quantization and FP16 conversion can change outputs relative to the original checkpoint.
• —Users are responsible for evaluating quality, safety, and suitability for their use case.

License

Apache-2.0, matching the upstream model artifacts. Please retain attribution to Qwen and Unsloth when redistributing this conversion. "Qwen" and related names are used only to identify the upstream model; this repository is not affiliated with or endorsed by Alibaba or Unsloth.