Ttimms/zaya1-8b-nvfp4-w4a4-uniform

ZAYA1-8B NVFP4 W4A4 — uniform

Methodology — how this build was pruned, quantized, and evaluated, and the silent failures found along the way: <https://github.com/t-timms/blackwell-16gb-moe>

## ⚠️ Does not load on a current transformers / vLLM stack (found 2026-09-03) If you are on `transformers` 5.x, this checkpoint will fail at engine init: `` rope_params = self.config.rope_parameters[layer_type] KeyError: 'hybrid' ` **Known-good:** the stack this was built and measured on — transformers 4.57.1, vLLM 0.20.2. **Confirmed broken:** transformers 5.15.1, vLLM 0.26.0. The exact version where it broke has not been bisected, so treat 4.57.1 as the only verified-working pin rather than assuming any <5 release works. **Cause.** transformers now ships a native zaya implementation. It defaults layertypes = ["hybrid"] * numhiddenlayers` and indexes `ropeparameters by layer type; this checkpoint carries the pre-refactor **flat** ropeparameters`. Being built from the pre-refactor 80-layer base — described below as a distinction of this work — is what causes it. **This is not fixable by editing `config.json`.** Nesting `ropeparameters under "hybrid" clears the KeyError and then fails structurally: upstream batches all 16 experts into one stacked tensor and fuses fc1 into w13, where this checkpoint stores them per-expert and unfused. 3,282 model parameters have no counterpart in the checkpoint. **The fix is a re-quantization from the current Zyphra/ZAYA1-8B**, which produces the modern layout natively. That work is prepared but not yet run, and a re-quantized checkpoint would be a **new artifact** — the accuracy and throughput figures on this card were measured on *this* artifact and would not carry over without being re-measured. Full analysis: [RESEARCH.md` §5.24](https://github.com/t-timms/zaya1-nvfp4-w4a4/blob/master/RESEARCH.md). Everything below this box was accurate on the stack it was measured on and is left unchanged.

6.02 GB. Every one of 1,320 Linears in packed NVFP4 W4A4 — 4-bit weights and 4-bit activations, zero BF16 exemptions — on native CUTLASS FP4 tensor-core kernels inside a 16 GB consumer Blackwell budget (RTX 5070 Ti, SM120). Vendor-official 4-bit releases — Google's Gemma 4 qat-w4a16, Zyphra's own ZAYA1-8B-MXFP4-Experts — are weights-only. This quantizes activations too.

Which one do I download?

The 9.46 GB build's 384 BF16 exemptions cost 3.44 GB and buy 0.71 pp of HellaSwag accuracy (95% CI [−1.26, −0.15], paired exact-binomial McNemar over 14,319 items). On a 16 GB card that is usually the wrong trade, which is why this one is the default. Both are maintained; neither supersedes the other.

Before you download

vllm serve Ttimms/zaya1-8b-nvfp4-w4a4-uniform --dtype bfloat16 --enforce-eager \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 5, "prompt_lookup_min": 2}'

Measured, `enforce_eager=True`, median of 5 process invocations: 9.52 tok/s single-stream, 73.4 tok/s at batch-8 (7.71×, 96% of ideal linear scaling). Accuracy: HumanEval 72.6% pass@1 [65.3, 78.8], GSM8K 65.5% [62.9, 68.0], MMLU-Pro 0-shot 48.1% [44.5, 51.8]. Earlier 104.7 / 105.3 tok/s figures were measured under CUDA graphs and are retracted — see the known-issue section.

Architecture

graph TD
    Base["Zyphra/ZAYA1-8B-legacy<br/>80-layer MoE - reasoning - Apache-2.0"]

    subgraph Build ["Quantization - RTX 5070 Ti, SM120"]
        Cal["llm-compressor NVFP4 W4A4 calibration<br/>4-bit weights + 4-bit activations"]
        Fork{"keep outlier-sensitive<br/>Linears at BF16?"}
    end

    subgraph HF ["Published checkpoints"]
        Mixed["zaya1-8b-nvfp4-w4a4 - 9.46 GB<br/>936 W4A4 + 384 BF16-exempt - eval control"]
        Uni["zaya1-8b-nvfp4-w4a4-uniform - 6.02 GB<br/>1,320 W4A4 - 0 exemptions"]
    end

    subgraph Serve ["Serving & evaluation"]
        vLLM["vLLM - native CUTLASS FP4 tensor-core kernels<br/>enforce_eager=True (CUDA graphs corrupt on SM120)"]
        Bench["HellaSwag acc 46.5% / 45.8%<br/>9.5 tok/s single - ~74 tok/s batch-8"]
    end

    Base --> Cal --> Fork
    Fork -- yes (384) --> Mixed
    Fork -- no --> Uni
    Mixed --> vLLM
    Uni --> vLLM
    vLLM --> Bench

What the two size figures actually measure

6.02 GB is this build's whole repository; the 9.46 GB figure quoted for the mixed build is its `model.safetensors` alone (its repository total is 9.49 GB). Both numbers are individually correct and both are already published, but they measure different things — the table above gives each basis explicitly. The reduction holds either way: −36.7% on weights, −36.6% on repository totals. Measured from the Hub, 2026-08-09.

⚠️ Known issue: CUDA graph capture corrupts output on SM120

Throughput figures previously published on this card (104.7 tok/s here / 407.4 tok/s batch-8 elsewhere in this project) were measured with CUDA graphs enabled — a code path confirmed 2026-08-14 to produce numerically wrong output on this hardware, independent of MoE backend. A sweep of flashinfer_cutlass (default), cutlass, and marlin (weight-only) all produced garbage under graph capture; only enforce_eager=True generated correctly. Since Marlin barely touches the FP4 MoE path and still failed, the fault is graph capture itself, not any one kernel. Two adjacent upstream issues exist — CUTLASS #3096 (different, non-graph-capture bug) and FlashInfer #2776 (graph-capture-specific, but its stated root cause doesn't explain Marlin failing too) — neither currently offers a fix that preserves CUDA graphs for this failure mode. See Pape, Evertz & Schönherr (arXiv:2605.19537) for the general phenomenon of backend-dependent correctness drift in LLM serving. Full sweep, coherence re-verification, and citations in RESEARCH.md and ROADMAP.md on the GitHub repo.

Set `--enforce-eager` (or `enforce_eager=True`) when serving this checkpoint. It is the only configuration confirmed to produce coherent output.

Throughput and memory (measured, enforce_eager=True)

Measured with vLLM's own benchmark CLI, not a bespoke harness. 5 separate process invocations per configuration, GPU otherwise idle:

vllm bench latency --model <checkpoint> --dtype bfloat16 --kv-cache-dtype fp8 \
  --gpu-memory-utilization <frac> --max-model-len 4096 --no-enable-prefix-caching \
  --input-len 128 --output-len 256 --batch-size <1|8> --enforce-eager \
  --num-iters-warmup 1 --num-iters 1

Throughput is still unchanged between checkpoints — the two land within run-to-run noise of each other at both batch sizes. That part of the original claim held; only the absolute numbers were wrong.

Batching is near-ideal (96–98% of theoretical linear scaling on batch-8), which is evidence the MoE decode step's per-step cost does not grow meaningfully with batch size.

The two checkpoints cannot be run at the same memory fraction on a 16 GB card. The mixed checkpoint fails at --gpu-memory-utilization 0.85 with ValueError: No available memory for the cache blocks, and at 1.0 with Free memory on device cuda:0 (14.66/15.92 GiB) ... less than desired because a desktop session holds ~1.3 GiB. Its working range is narrow; the uniform build runs at 0.85 with headroom — the practical case for the smaller checkpoint, more so than the accuracy difference.

Retracted, do not cite: 104.7 / 105.3 tok/s single-stream, 102.6 / 407.4 tok/s from elsewhere in this project (all measured under CUDA graphs).

An unverified external signal exists and disagrees with the table above. llama.cpp PR #23112's own author reports 45.9 tok/s on a slower RTX 4070 Ti (Q4KM GGUF) — beating this checkpoint's 9.5 tok/s by ~4.8×. Five attempts to reproduce it on this project's own SM120 hardware hit the same non-deterministic hang each time, ruling out the model and toolkit version as causes; looks like a WSL2/driver-level issue. Not resolved as of 2026-08-14 — see GitHub repo RESEARCH.md §5.16 for the full diagnostic log. Update, same day: the gap likely has a principled cause independent of the hang above. Activation quantization (W4A4) gives no speed benefit at batch-1 — decode there is memory-bandwidth-bound, and quantizing activations only helps when compute is the bottleneck. Weight-only quantization is expected to win at batch-1 by design; this checkpoint's advantage is memory footprint and batched throughput (see the batch-8 row above). Detail: RESEARCH.md §5.17. A real batch-1 lever that does exist: vLLM's built-in n-gram speculative decoding gives a validated 2.2× speedup on coding-edit prompts (zero training, one config flag) — no gain on free-form generation, as expected. Deployed to the production serve script and validated live through the real OpenAI-compatible API. Detail: RESEARCH.md §5.18.

This or one of the GGUF builds? Read this first

The most-downloaded ZAYA1-8B quantizations are GGUF k-quant ladders (`Abiray/ZAYA1-8B-GGUF`, `JusteLeo/ZAYA1-8B-GGUF`). Those are weights-only; this is W4A4 — weights and activations. Pick on how you serve, not on bit-width:

Use a GGUF build if you run one conversation at a time, are not on Blackwell, or want a size ladder to fit a specific VRAM budget. Use these if you serve with vLLM on a 50-series card, want throughput under concurrency, or need a checkpoint whose accuracy has actually been measured.

On single-stream speed: the GGUF builds are probably faster, and that is expected

Neither GGUF repo publishes a throughput number, so no measured head-to-head exists. The only figure in circulation comes from PR #23112's own author: 45.9 tok/s on an RTX 4070 Ti with Q4KM — a slower card than the RTX 5070 Ti used here, at roughly 4.8× the 9.5 tok/s measured on this checkpoint. Five attempts to reproduce it on this project's hardware hit the same non-deterministic hang (WSL2/driver-level, not model-level), so treat it as unverified — but it is not disputed here.

There is a principled reason to expect it to hold — though not the usual one. Batch-1 decode is not compute-bound, so quantizing activations has no bottleneck to relieve; W4A4's return arrives under batching, which is where the batch-8 figure above comes from.

But it is not bandwidth-bound either, and that part is worth stating precisely. ~760 M active params at ~4.5 effective bits (NVFP4 + FP8 block scales) is ~0.5 GB of weight traffic per token; at this card's 896 GB/s that is a ~0.56 ms floor — about 1,790 tok/s. The measured 9.5 tok/s is 105 ms/token, or ~0.5% of that roofline. The GPU is idle for essentially the whole decode step, so the real limit is per-step dispatch overhead across 80 sequential layers under the mandatory --enforce-eager, not the weight format. The reported llama.cpp figure sits at ~4.6% of its own roofline — also overhead-bound, just ~9× less so, which explains the gap better than bit-width does.

So a weight-only quant winning at batch-1 is expected, but how much of the gap is the scheme and how much is the missing CUDA graphs is an open question, not a settled one. scripts/sweep_cudagraph_modes.sh in the GitHub repo tests the five vLLM graph modes, with a coherence gate so that no mode can report a throughput number for output nobody verified. On footprint, the fair size-matched comparison to Q5KM (6.43 GB) is the `-uniform` build at 6.02 GB rather than the 9.46 GB mixed build.

If single-stream latency is what you care about, take a GGUF build — and note that `enable_thinking=False` will do far more for your latency than any quantization choice, at a cost this card quantifies and the GGUF cards do not.

Accuracy: measured, paired, and honest about its limits

The two checkpoints were compared with a paired exact-binomial McNemar test on discordant items, joined per doc_id, over 14,319 items each across four pure-loglikelihood tasks. No generation is involved, so this is immune to the <think>-never-terminates artifact that invalidates naive generative evals on this model. No chat template — these are ranked-continuation tasks.

Read the intervals, not the p-values. No comparison survives Bonferroni correction (α = 0.05/7 = 0.0071). That is absence of resolution, not evidence of absence. HellaSwag is the only adequately powered task here, and its 95% CI excludes zero. The three smaller benchmarks cannot rule out regressions of −4.97 pp (arcchallenge `accnorm`) or −6.04 pp (winogrande). Five of seven comparisons point negative, consistent with a small real regression that only HellaSwag has the samples to detect. Defensible claim: −0.71 pp on HellaSwag for −36% checkpoint size. Anything stronger is unsupported by this data.

An earlier ARC-Easy comparison suggested no cost at all. That was an unpaired two-proportion test applied to paired data: both checkpoints are quantizations of one base model scored on the same items, and discarding that correlation discards the statistical power. It also happened to favour the wrong checkpoint. Aggregate accuracy output cannot be converted into a paired test after the fact — it requires re-running the model with per-item logging.

Reproduce with scripts/run_phase_a.py and scripts/analyze_phase_a.py in the GitHub repo.

Generative benchmarks (2026-08-16)

The tests above are loglikelihood — they score pre-written continuations, so they measure ranking, never producing. These three measure generation:

enforce_eager=True, thinkbudget 4096, temp 0.6 / topp 0.95 (Zyphra's published recommendation), seed 42. Reproduce with scripts/run_budget_forced_suite.sh.

On HumanEval: published figures put Qwen 3 7B at roughly 68–72% and Llama 3 8B at 62–65% — both at full precision. This checkpoint matches or exceeds them at 4-bit weights and activations in 6.02 GB. Zyphra publishes no HumanEval number for ZAYA1, so there is no retention figure to quote; this appears to be the first measurement of it at any precision.

On MMLU-Pro — read this before comparing to Zyphra's 74.2%. That number is not a valid comparison point for this one. lm-eval's MMLU-Pro task is 5-shot; this harness is 0-shot, because ZAYA1's <think> trace frequently never terminates and the standard harness scores the unterminated trace as the answer (the artifact that put IFEval at 19.8% against an 85.58% reference). Zyphra's figure also comes from a private harness with undisclosed generation limits. The gap is a protocol difference, not quantization damage: published INT4 loss on MMLU-Pro is ~1.6 pp, and the paired test above bounds this checkpoint's cost at −0.71 pp HellaSwag.

The budget was tested, not assumed. Both benchmarks were re-run at think_budget 8192 and compared with paired McNemar on identical items — GSM8K +0.15 pp (p=0.9581), MMLU-Pro +3.29 pp (p=0.0673). Neither is significant, so 4096 is the correct operating budget and more reasoning room does not help this model. Detail: RESEARCH.md §5.22.

On GSM8K: honest but unremarkable — Llama 3.2 3B reports 77.7% at 8-shot. This run is 0-shot, 78% of traces truncate, and GSM8K is heavily contaminated. Treat it as a regression check that quantization did not break arithmetic reasoning, not as evidence of mathematical ability.

⚡ enable_thinking=False — 8.5× faster, but it costs 17–29 accuracy points

ZAYA1's chat template (Zyphra's, not ours) supports disabling the reasoning trace, which pre-closes <think> so the model answers immediately. This is the single biggest latency lever available, and it is not free. Measured on this checkpoint, paired McNemar on identical items:

*Read this before deciding how to serve the model: ZAYA1's accuracy is its reasoning, and its reasoning is what makes it slow — the two cannot be separated.* It is not a fast interactive model, and no serving configuration makes it one. If you need low latency more than accuracy, a weight-only quant of a non-reasoning model will serve you better.

Where the flag is genuinely useful: per-request routing, not a global switch. vLLM accepts it per request:

{"model": "...", "messages": [...],
 "chat_template_kwargs": {"enable_thinking": false}}

Send mechanical work (file reads, trivial edits, simple tool calls) down the fast path and keep reasoning for real problem-solving. Note that *12 / 53 / 173 items respectively were solved only with thinking disabled* — some tasks are actively hurt by overthinking, so routing can win twice.

Full analysis: `RESEARCH.md` §5.23.

Why the exemptions were removable

The 9.46 GB checkpoint exempts 12 MoE layers whose calibrated activation max_abs exceeds 500 — the worst being 8,896 at L75.experts.1.linear_fc2, 622× the median. Only 24 linear_fc2 modules actually offend, but FusedMoE requires uniform quantization per layer, so protecting them forces exemption of fc1 and fc2 across all 16 experts in each affected layer: 384 Linears, 3.44 GB. A 16× overpay, where 16 is num_experts.

The exemptions turned out to be largely redundant. SOAR — which targets the same FP8 block-scale rounding error the outliers cause — was introduced after the mixed-precision decision, and the combination was never re-tested. The worst channel in the model quantizes coherently at W4A4 without them.

No residual correction (ARCQuant or otherwise) is applied to this checkpoint, and none is required. The numbers above are for the artifact exactly as published.

Base model revision — read before reproducing

Quantized from the original 80-layer ZAYA1-8B config (num_hidden_layers: 80, moe_router_topk, zaya_use_eda / zaya_use_mod, transformers 4.57.1).

In late June 2026 Zyphra refactored ZAYA1-8B into upstream-transformers form: Zyphra/ZAYA1-8B now reports num_hidden_layers: 40 with layer_types: hybrid and num_experts_per_tok, and the original moved to `Zyphra/ZAYA1-8B-legacy`.

Core dimensions are unchanged (hidden 2048, 16 experts, top-1 routing, vocab 262272), so this is a re-expression rather than a new model — but reproduce against `Zyphra/ZAYA1-8B-legacy`, or you will hit an architecture mismatch.

Verified, not just inferred (2026-08-14): fetched model.embed_tokens.weight directly from both repos via HTTP range request (no full download) and compared the raw bytes — byte-for-byte identical, 1,074,266,112 bytes, BF16, shape [262272, 2048]. The two repos' safetensors index files also report an identical aggregate total_size (17,680,978,928 bytes) despite the tensor layout changing from 2,483 named tensors (legacy) to 1,283 (current, fused/batched expert tensors, transformers-conventional naming). This is why base_model lists both repos below with base_model_relation: quantized — not an assumption, a direct measurement on the one tensor that could be checked without reverse-engineering the expert-fusion mapping. Every other published NVFP4/GGUF/BNB quantization of this model tags only the current Zyphra/ZAYA1-8B — this checkpoint is the only one built from and verified against the pre-refactor structure directly.

Quantization details

quantization_manifest.json carries the full machine-readable config, the exact outlier-layer list, and the evaluation block reproduced above.

Usage

Requires vLLM built from source with SM120 NVFP4 CUTLASS kernels — stock wheels do not compile them. Inference must run in bfloat16.

# vLLM built from source with: TORCH_CUDA_ARCH_LIST=12.0
vllm serve Ttimms/zaya1-8b-nvfp4-w4a4-uniform --dtype bfloat16 --enforce-eager \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 5, "prompt_lookup_min": 2}'

The exact command above is `scripts/serve.sh` in the repo.

Serving notes for this model family:

• —enable_prefix_caching=False is required — ZAYA's CCA state is not cacheable, and vLLM defaults this to True.
• —`--enforce-eager` is required for correct output, not just recommended — CUDA graph capture corrupts generation on this card regardless of MoE backend. See "Known issue" above before serving without it.
• —`--speculative-config` is optional but free — lossless n-gram speculative decoding, validated 2.2× faster on coding-edit prompts (no gain on free-form generation, since there's no prompt/output overlap to exploit). Detail: RESEARCH.md §5.18.
• —Set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 on a 16 GB card; the estimator otherwise reserves 3.5+ GiB and pushes the KV cache into OOM.
• —Generative evals need budget forcing. ZAYA is a reasoning model whose <think> trace frequently never terminates within a normal token budget, and it answers in \boxed{} form — stock harnesses score it near random for reasons unrelated to quantization.

Why this is hard

ZAYA1-8B is an 80-layer MoE (760M active / 8.4B total) with Zyphra's CCA (compressed convolutional attention). W4A4 requires calibrating activation scales, not just weights; the compressed-tensors calibration path has a silent NaN-producing trap if you calibrate through the fake-quant nn.Linear.forward wrapper; and the NVFP4 global-scale convention (2688 / max_abs, divisor form, block scales pre-multiplied) is undocumented — getting it wrong produces silent pad-token collapse rather than an error.

Known limits

• —Single-stream and batch-8 throughput are now both measured for both checkpoints with vllm bench latency under enforce_eager=True (see above): 9.52 tok/s single-stream / 73.4 tok/s batch-8 here, essentially identical to the 9.46 GB build. The earlier CUDA-graph figures (104.7/105.3 single, 102.6/407.4 elsewhere) are retracted — see "Known issue" above.
• —No BF16 control eval exists — a BF16 ZAYA1-8B needs ~35 GiB and does not fit the source hardware. All accuracy claims here are relative to the 9.46 GB checkpoint, not to full precision.
• —The three smaller benchmarks are underpowered, as stated above.

License

Apache 2.0 — matches the ZAYA1-8B upstream license.