digi-texx/Qwen3.8-27B-FULL-NVFP4

Qwen3.8-27B FULL-NVFP4

Every linear layer in NVFP4, including `lm_head`.

`QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4` quantizes the transformer body to NVFP4 but leaves lm_head in BF16 — it is listed in quantization_config.ignore. For a 248,320-token vocabulary that single tensor is 2.54 GB, a large share of what remains unquantized.

This model quantizes it, to 0.715 GB — a 3.55× reduction on that tensor and ~1.8 GB off the checkpoint — using MR-GPTQ for the weights and H-Scale for the block scales, both calibrated on activations captured from the already-quantized backbone.

Nothing else is changed. The body tensors are bit-identical to the base model, and rope_parameters / max_position_embeddings are the base model's own values.

Method

lm_head is quantized to NVFP4 with a group size of 16: per-group E4M3 block scales under a per-tensor FP32 global scale, matching the format the rest of the checkpoint already uses.

1. 1.Calibration activations are captured from the real W4A4 backbone. The inputs lm_head actually receives at inference come from a body that quantizes activations at each of its 64 layers. Calibrating against a BF16-decompressed body would fit a distribution the deployed model never produces, so the activations were captured from the quantized model itself, at the exact tensor lm_head consumes (post-final-norm). The capture was verified end to end: feeding the last captured token through the original BF16 lm_head reproduces the token the model itself generated.
2. 2.MR-GPTQ (arXiv:2509.23202) — a weight-domain scale search (100-step shrink line search minimising Σ|x−x̂|^2.4) followed by GPTQ's Cholesky error propagation across input channels, using a Hessian H = Σ xxᵀ accumulated over the captured activations.
3. 3.H-Scale (arXiv:2608.28113) — per group, a search over K=16 hardware-valid E4M3 candidate scales (6 up, 9 down the E4M3 ladder), keeping the one that minimises Σⱼ hⱼ (wⱼ − ŵⱼ)² with h = diag(XᵀX) from the same activations.

Measured, on this vocabulary

Relative reconstruction error of lm_head's output against the BF16 reference under true W4A4 (weights and activations quantized), on held-out tokens never used for calibration:

Two caveats worth stating. H-Scale contributes almost nothing on top of MR-GPTQ (0.067387 → 0.067380); the two overlap because GPTQ-style error propagation already absorbs most scale-selection error. And with exact BF16 weights the W4A4 error is still 0.055496 — 70% of the total is activation quantization, which no weight-side method addresses.

Calibration data

The exact calibration sample is published as `digi-texx/calib-agentic-sample-Qwen3.8-27B-QUASAR-NVFP4` — not a representative extract, but the documents the quantizer actually saw, with per-row provenance. The full lineage:

11 public agentic / tool-calling datasets
  -> digi-texx/calib-agentic-normalized     6,044,537 rows   unified schema
  -> digi-texx/calib-agentic-curated        5,835,723 rows   dedup + quality filters
  -> digi-texx/calib-agentic-sample-...           145 rows   target 50% tool-calling
  -> 65,536 tokens                                           consumed by the quantizer

Curation dropped 208,814 rows: 107,937 near-duplicates (MinHash/LSH at 0.85), 61,217 degenerate, 30,275 empty or trivial, 9,371 exact duplicates, 14 orphan tool results.

145 documents, 448,199 tokens after rendering with the model's own chat template; the quantizer consumed the first 65,536 tokens of a seeded shuffle of those documents.

55.2% of the sampled documents contain tool calls, by construction — the mixture was drawn to a target tool-calling fraction.

Data licensing

Contributing datasets are permissive (CC-BY-4.0, Apache-2.0, MIT, ODC-BY) with one exception: `Salesforce/APIGen-MT-5k` is CC-BY-NC-4.0 (NonCommercial) and contributes 1 document / 4,737 tokens (1.1% of the corpus). Anyone with commercial requirements should evaluate whether a NonCommercial-licensed calibration sample of that size affects their intended use.

Chat template

This model ships a reworked chat template (qwen3.8-unified-v4) in place of the base model's, in both chat_template.jinja and the chat_template field of tokenizer_config.json — kept byte-identical, since loaders differ in which one they read and a mismatch would make behaviour depend on the loader rather than the model.

It began (v1) as a unified merge of the fixes from three community Qwen chat-template repositories:

• —`Moore2877/Qwen-Fixed-Chat-Templates-llamacpp`
• —`froggeric/Qwen-Fixed-Chat-Templates`
• —`Jaidchen/Focus-Red`

Where those repositories disagreed, the prefix-KV-stable resolution was taken, because the target workload is multi-turn agentic serving where prefix cache reuse dominates. Concretely, the reasoning-effort instructions stay in the system block rather than being appended to the latest user message: appending them to loop.last looks like it protects the prefix, but in multi-turn chat it destroys it — turn 1 carries the appended text, then stops being loop.last at turn 2, so its tokens change and every cached turn is invalidated.

Version history

• —v2 ported two fixes from froggeric's later v22.5 release, verified byte-for-byte against it: OpenAI-style video_url content items (previously raised Unexpected item type), and a fully depth-aware JSON tool-response truncator. Froggeric's own truncator (and every other fork checked) either skips truncating JSON entirely or head/tail-splices it — which stitches a real closing }]} onto a truncated body, so a 400-row result can read as complete with 380 rows silently missing. This template's truncator tracks string/escape/container-depth state, cuts at the last element boundary inside the budget, and closes the containers itself, so truncated output still parses as JSON and carries a machine-readable "__truncated__" sentinel.
• —v3 adopted three behaviours distilled from further community forks (mdrbal83's Qwen3.8-Sharp-Seeded-Chat-Template, peculiar-ragdoll's Qwen-Sharp-Chat-Templates, and OliviaRossi's Improved-Chat-Template-for-Qwen-3.x), all exposed as chat_template_kwargs and all default-on:
• —terse — appends a conciseness system-prompt block (no preamble, lead with the answer).
• —automatic [TOOL_REQUEST] detection — stands down this template's own tool-call instructions when the serving runtime has already injected its own protocol (a real LM Studio/MLX issue; inert under SGLang, which never emits that marker).
• —think_seed — opens the reasoning block with <think>\nLet me instead of <think>\n, skipped on the turn immediately following a <tool_response>. This targets a documented Qwen3.8 failure mode (a clipped "We need answer" note-taking style that can run away past 8k reasoning tokens); the upstream fork's own benchmark claims a 40-74% reduction in mean reasoning tokens with no accuracy loss, but those numbers are third party and have not been independently reproduced against this checkpoint. Treat the mechanism as plausible, the magnitude as unverified.
• —v4 renders the <tools> block sorted by function name rather than in caller-supplied array order. Tool order carries no semantics to the model, but the tools block sits at the very start of the prefix — two calls offering the same tool set in a different order (e.g. different agents or phases in a multi-agent harness assembling the same tools independently) previously rendered a different token sequence at position 0 and evicted the whole session's radix-cache entry for a difference the model never needed to see. Sorting is a no-op when the caller is already consistent.

Verified properties (checked against this template on every release since v1)

• —Appending a new turn leaves the entire previous prefix byte-identical — a 2-turn and 3-turn rendering of the same conversation share a common prefix with zero divergence — so multi-turn cache reuse holds.
• —Changing reasoning_effort mid-conversation does invalidate the prefix (diverges within the first ~50 characters of the system block). This remains a deliberate, unfixed trade: effort changes turn-to-turn in agentic harnesses that route effort by task phase, and moving the steering text out of the system block to protect the cache would place a system-role message mid-conversation — a placement Qwen3.8 was not trained on. Left as-is pending a measured A/B rather than an unverified structural change.
• —New in v4: reordering the same tool set no longer invalidates the prefix at all (previously it diverged at position 0, i.e. the entire prompt).

It is the template the calibration corpus was rendered with, so the activations the quantizer saw match the format the model is served with. It targets agentic and tool-calling use, and exposes options that can be passed through chat_template_kwargs:

It also classifies tools as mutating vs read-only and can nudge on repeated or ping-ponging tool calls. The base model's template remains valid input to the model; this one is what the calibration data was rendered through.

Usage

Loads anywhere compressed-tensors NVFP4 is supported, the same as the base model — the head is stored as weight_packed / weight_scale / weight_global_scale / input_global_scale, and lm_head has been removed from quantization_config.ignore.

Note. Some inference stacks assume lm_head is never quantized: a ParallelLMHead is not a LinearBase, so a quantization method may not be resolved for it and the packed weights can be read as though they were unquantized — which produces silent garbage logits rather than an error. If you see incoherent output, verify that your runtime resolves a quantization method for lm_head rather than assuming it loaded correctly.

Limitations

• —Calibration is agentic/tool-calling/code-weighted. A very different serving distribution may prefer a different calibration mixture.
• —The vision tower, MTP layers, and embeddings remain in the base model's ignore list; only lm_head was added.
• —Quantization error concentrates where the calibration data is thin. Behaviour on domains far from the mixture above is not characterised here.

Citation

arxiv.org/abs/2608.13966

@article{counathe2026quasar,
  title={QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction},
  author={Counathe, Vincent and Athiwaratkun, Ben and De Sa, Christopher and Zhang, Tianyi},
  journal={arXiv preprint arXiv:2608.13966},
  year={2026}
}

Quantization methods used for lm_head:

@article{egiazarian2025mrgptq,
  title={Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization},
  author={Egiazarian, Vage and others},
  journal={arXiv preprint arXiv:2509.23202},
  year={2025}
}

@article{hscale2026,
  title={H-Scale: Hessian-Guided Scale Refinement for NVFP4 Sub-Byte LLM Inference},
  journal={arXiv preprint arXiv:2608.28113},
  year={2026}
}