carlopires/Qwopus3.8-27B-Flash-PTQ-NVFP4-NInfer
Qwopus3.8-27B-Flash PTQ NVFP4 for NInfer
This release packages Qwopus3.8-27B-Flash for NInfer using post-training quantization (PTQ) to NVFP4. It did not undergo quantization-aware training (QAT).
QUASAR references in the conversion documentation identify tooling and runtime compatibility, not a QAT training history. They do not establish that this artifact reproduces the quality of a QUASAR-trained checkpoint.
The downloadable format is NInfer's native .ninfer container. See the provenance, measured performance, and evaluation limitations below for the scope of the release and its validation.
Qwopus3.8-27B-Flash is an agent-oriented "Flash" fine-tune of Qwen/Qwen3.8-27B published by Jackrong as GGUF only. The source checkpoint has no public Safetensors distribution, so this artifact was built by converting the publisher's Qwopus3.8-27B-Flash-MTP-BF16.gguf to a canonical BF16 HF checkpoint, applying post-training NVFP4 quantization with QUASAR-compatible tooling, and converting to the NInfer container.
The artifact is intended only for NInfer. It is not a Transformers checkpoint, Safetensors distribution, GGUF file, or generic NVFP4 interchange format. NInfer selects it with --model qwopus3.8-27b (this fork's default); the published MirkoCovizzi/Qwen3.8-27B-QUASAR-NVFP4-NInfer artifact remains available with --model qwen3.8-27b.
Artifact
The file contains the Text backbone, Vision tower, MTP draft model, optimized proposal head, tokenizer, chat template, generation configuration, and media processor resources required by NInfer.
Verify the download with:
printf '%s %s\n' \
'4e8d9236b8a9c8383a1168d1f55a1804825774ed09b42f34b3c23a57177959c8' \
'qwopus3_8_27b_nvfp4.ninfer' | sha256sum --checkRequirements
- 64-bit Linux;
- an NVIDIA Blackwell GPU with FP4 support;
- CUDA Toolkit 13.1 or newer;
- NInfer with
qwopus3.8-27bregistered, from carlopires/ninfer-rtx5090-mobile, commit `830e26bb`.
The artifact used 15.31 GiB of GPU weight storage for Text-only execution and 16.06 GiB with MTP materialized during validation. Context capacity requires additional memory. A 24 GiB RTX 5090 Laptop GPU supports useful contexts with INT8 KV; a 32 GiB desktop RTX 5090 provides more context headroom and is NInfer's primary performance target.
Download and run
Build the runtime:
git clone https://github.com/carlopires/ninfer-rtx5090-mobile.git
cd ninfer-rtx5090-mobile
git checkout 830e26bb34504640d0e7fabcd203b1e7a81d60ff
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -jDownload the artifact:
hf download carlopires/Qwopus3.8-27B-Flash-PTQ-NVFP4-NInfer \
qwopus3_8_27b_nvfp4.ninfer \
--local-dir modelsRun the CLI with MTP speculative decoding:
./build/apps/ninfer models/qwopus3_8_27b_nvfp4.ninfer \
--model qwopus3.8-27b \
--prompt "Explain quantization-aware training in three sentences." \
--max-context 4096 --max-new 256 \
--kv-dtype int8 \
--spec mtp --draft-tokens 3 --lm-head-draft--model qwopus3.8-27b is this fork's default and may be omitted. Loading the published Qwen3.8 QUASAR artifact on the same build requires --model qwen3.8-27b; a mismatched selector is rejected with an identity error.
Run the OpenAI/Anthropic-compatible server:
./build/apps/ninfer-serve models/qwopus3_8_27b_nvfp4.ninfer \
--host 127.0.0.1 --port 18080 \
--max-concurrency 1 --max-context 4096 \
--kv-dtype int8 \
--spec mtp --draft-tokens 3 --lm-head-draftIncrease context and concurrency only within the available VRAM. NInfer fixes active-request capacity at startup.
Supported NInfer features
This is a complete Qwen3.8 artifact rather than a Text-only weight file. The registered route supports:
- thinking and non-thinking text generation;
- image, multi-image, video, and mixed multimodal messages;
- MTP speculative decoding with optimized proposal-head lookup;
- BF16 and INT8 group-64 KV cache;
- CUDA Graph decode and compatible-prefix reuse;
- the NInfer CLI and bounded concurrent server;
- OpenAI Responses Core, OpenAI Chat Completions, and Anthropic Messages translation.
NInfer is a single-GPU engine. It does not provide CPU/GPU offload, multi-GPU execution, distributed serving, or large-scale preemptive continuous batching.
How it was built
Qwopus3.8-27B-Flash publishes GGUF weights only, so the conversion ran in three stages: GGUF to canonical BF16 HF checkpoint, llmcompressor NVFP4 quantization with the QUASAR fusion repair, and the standard NInfer QUASAR NVFP4 converter against the resulting nvfp4-pack-quantized checkpoint.
Stage 1 — GGUF to BF16 HF
Qwopus3.8-27B-Flash-MTP-BF16.gguf (arch qwen35, 65 blocks = 64 text layers + 1 MTP layer, 16 full-attention + 48 Gated DeltaNet layers) was exported with tools/convert/qwen3_8_27b/gguf_export.py, which applies the inverse transforms of llama.cpp's Qwen3.5 mapping: GDN V-row reorder inversion, out_proj column inverse, conv1d channel inverse, ssm_a = -exp(A_log) recovery, and the llama.cpp norm +1 offset reversal. The result is 1,199 BF16 tensors: 866 Text/MTP tensors plus the 333 model.visual.* tensors from the fixed official Qwen3.8 checkpoint (Qwopus shares the base vision tower).
Why some technical identifiers still contain QUASAR
The public model name uses PTQ to describe how this artifact was prepared. Existing QUASAR-named converter and runtime identifiers (the Qwen38Nvfp4Quasar runtime profile, the qwen3_8_27b_quasar_nvfp4-v2 conversion recipe, the QUASAR scale math, and the nvfp4-pack-quantized compatibility contract) remain unchanged because they identify a compatibility contract, not the training method of every artifact that uses that contract.
Stage 2 — post-training NVFP4 quantization with QUASAR-compatible tooling
Stage 2 is a post-training quantization of the reconstructed Qwopus BF16 fine-tune, not a QUASAR-trained release.
llmcompressor 0.13 quantized all 496 Text Linear matrices of the reconstructed Qwopus BF16 checkpoint to nvfp4-pack-quantized (W4A4, group-16, static-local activations) — exactly the same 496-matrix set as the published QUASAR checkpoint. llmcompressor quantizes per nn.Linear, but NInfer's QUASAR profile requires a shared weight_global_scale across fusion groups (GDN in_proj_qkv+in_proj_z -> 16,384 rows; full-attention q+k+v -> 14,336 rows). tools/convert/qwen3_8_27b/fuse_nvfp4_groups.py re-quantized each fusion group from BF16 with the exact QUASAR scale math (global_scale = 2688/absmax, weight_scale = E4M3(max_abs/6 * global_scale), E2M1 codes) and was verified byte-exact against the published checkpoint (global scale 6171.55 for layer-0 qkv+z, matching QUASAR's 6171.55). The remaining 844 BF16 controls (norms, A_log, conv1d, MTP, Vision) were merged from the full-precision source and packaged into a six-shard nvfp4-pack-quantized checkpoint with the multimodal config shape.
Provenance review of the quantized source
The generated conversion report carries the QUASAR-QAT repository in its source.quantized fields. Local evidence confirms this is inherited converter metadata, not the weight source: the llmcompressor run log (nvfp4_full.log, 2026-09-04) records calibration over 65 blocks of the locally reconstructed Qwopus BF16 checkpoint (model_hf_full), and the converted checkpoint at /storage/qwopus/qwopus_nvfp4_sharded carries the Qwopus-reconstructed config (matching the fine-tune's merged output) rather than the upstream QAT config. The immediate packed-NVFP4 input was this locally produced checkpoint, which has no Hub repository or revision; the conversion report's repository/revision fields are preserved byte-for-byte above as the converter emitted them, with this note as the correction. The fine-tune itself enters the pipeline through Jackrong's BF16 GGUF.
Stage 3 — NInfer conversion
The standard Qwen3.8 NVFP4 converter detects the source's exact nvfp4-pack-quantized configuration and selects the dedicated QUASAR profile (Qwen38Nvfp4Quasar). NInfer preserves the packed E2M1 codes, E4M3FN block scales, FP32 weight divisors, and FP32 input divisors of the 400 source matrices supported by NInfer's NVFP4 execution leaves. Fusing matrices at the execution-consumer boundary produces 256 stored NVFP4 parents:
The remaining 96 QUASAR linears are the Gated DeltaNet in_proj_a and in_proj_b matrices with output extent 48, outside NInfer's registered NVFP4 execution geometry. The converter decodes each represented weight, rounds it to BF16, and fuses A then B into one [96,5120] control parent. This explicit NVFP4-to-BF16 cast is the only numerical conversion applied to QUASAR-format Text matrices.
The fixed official Qwen3.8 BF16 checkpoint supplies the two W8 vocabulary endpoints, Vision, the optimized draft head, and the six frontend resources. MTP and Vision use NInfer's registered groupwise storage allocation. The resulting tensor allocation is:
Reproduce the conversion
Download the pinned inputs:
hf download Qwen/Qwen3.8-27B \
--revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 \
--local-dir models/Qwen3.8-27B
hf download Jackrong/Qwopus3.8-27B-Flash-GGUF \
Qwopus3.8-27B-Flash-MTP-BF16.gguf \
--revision 3c24631f3a14fb86682bc0c60224e63d42dd5bf1 \
--local-dir models/qwopusCheck out the converter and install its Python dependencies in a Python 3.11 environment:
git clone https://github.com/carlopires/ninfer-rtx5090-mobile.git
cd ninfer-rtx5090-mobile
git checkout 830e26bb34504640d0e7fabcd203b1e7a81d60ff
python3.11 -m venv .venv
.venv/bin/python -m pip install torch numpy safetensors gguf==0.19.0 llmcompressor==0.13Export and quantize:
# GGUF -> canonical BF16 HF (Text/MTP tensors)
PYTHONPATH=. .venv/bin/python -m tools.convert.qwen3_8_27b.gguf_export \
--gguf models/qwopus/Qwopus3.8-27B-Flash-MTP-BF16.gguf \
--out models/qwopus/hf_bf16_full
# merge Vision tensors + frontend resources into the full HF directory, then:
# llmcompressor one-shot NVFP4 quantization of the 496 Text Linears, followed by
# tools/convert/qwen3_8_27b/fuse_nvfp4_groups.py to restore shared fusion-group
# global scales, and packaging into the six-shard nvfp4-pack-quantized checkpoint.Convert:
PYTHONPATH=. .venv/bin/python -m tools.convert.qwen3_8_27b.convert_nvfp4_quasar \
--model models/qwopus/model_hf_full \
--quantized-model models/qwopus/qwopus_nvfp4_sharded \
--out qwopus3_8_27b_nvfp4.ninfer \
--device cuda \
--model-id qwopus3.8-27bThe converter validates both checkpoint configurations, every selected source tensor and dtype, all 496 NVFP4 code/scale/divisor records, fused-group divisor equality, frontend resources, the complete 1,268-object plan, and the optimized proposal-head ranking input before opening the output.
Measured performance
Validation was performed on an NVIDIA GeForce RTX 5090 Laptop GPU (sm_120, 24 GiB), CUDA 13.1, NInfer commit 830e26bb, INT8 group-64 KV, 300-token greedy decode (see the fork's docs/gguf-conversion-feasibility-findings.md):
These are limited spot measurements from a single session on one prompt (2,572-token prefill, 300-token greedy decode, INT8 group-64 KV). They must not be extrapolated to all prompts, context lengths, or workloads, and they are not the broad benchmark corpus behind the published Qwen3.8 numbers. On these spot measurements the NVFP4 conversion decodes about 1.63× faster than the groupwise Qwopus conversion with MTP enabled, and its decode-without-MTP rate is close to the one spot-measured NVFP4 Qwen3.8 figure; the large prefill difference in the table reflects the different prompt length of that earlier measurement, not a controlled comparison.
Validation and quality caveats
Quantization and evaluation scope
This is a PTQ release, not a QAT release. Results published for the upstream fine-tune or other quantized deployments are not measurements of this artifact. Its quality must be assessed separately.
Validation performed
- the artifact loads and generates coherent text on the RTX 5090 Laptop through the public Engine route, including MTP=3 speculative decode;
- C++ profile resolution, the complete 1,268-object load plan, and converter preflight all passed.
A successful load and coherent responses do not demonstrate benchmark quality. No broad accuracy or long-context evaluation has been run on this converted artifact, and Qwopus publishes no HF Safetensors ground truth for a numeric comparison against the original fine-tune.
MTP acceptance
The reported spot check observed approximately 41% MTP acceptance. The upstream publisher reported 80.7% in a different evaluation. These figures are not a controlled comparison, and acceptance is not an answer-accuracy score. This comparison alone does not show that PTQ caused the difference.
Provenance
See qwopus3_8_27b_nvfp4.ninfer.conversion.json for the generated conversion report and artifact-manifest.json for a compact machine-readable summary.
Note on the generated report: its source.quantized fields name the upstream QAT repository because the converter inherits the QUASAR compatibility profile's metadata; the actual quantized input was the locally produced PTQ checkpoint described under "Provenance review of the quantized source" above. The report is preserved byte-for-byte as an audit record, with this note and the manifest's quantization_provenance annotation as the correction.
Migration note
Previously published as Qwopus3.8-27B-Flash-QUASAR-NVFP4-NInfer. The rename clarifies PTQ provenance; the model artifact and runtime identifiers are unchanged.
License
This artifact is distributed under the Apache License 2.0, matching the Qwen3.8 base model and the Qwopus3.8-27B-Flash source repository. Users remain responsible for complying with the source licenses and applicable laws. If this artifact is useful, cite the original Qwen3.8 model and Jackrong's Qwopus3.8-27B-Flash work.