Ttimms/zaya1-8b-nvfp4-w4a4

ZAYA1-8B NVFP4 W4A4

## ⚠️ Does not load on a current transformers / vLLM stack (found 2026-09-03) If you are on `transformers` 5.x, this checkpoint will fail at engine init: `` rope_params = self.config.rope_parameters[layer_type] KeyError: 'hybrid' ` **Known-good:** the stack this was built and measured on — transformers 4.57.1, vLLM 0.20.2. **Confirmed broken:** transformers 5.15.1, vLLM 0.26.0. The exact version where it broke has not been bisected, so treat 4.57.1 as the only verified-working pin rather than assuming any <5 release works. **Cause.** transformers now ships a native zaya implementation. It defaults layertypes = ["hybrid"] * numhiddenlayers` and indexes `ropeparameters by layer type; this checkpoint carries the pre-refactor **flat** ropeparameters`. Being built from the pre-refactor 80-layer base — described below as a distinction of this work — is what causes it. **This is not fixable by editing `config.json`.** Nesting `ropeparameters under "hybrid" clears the KeyError and then fails structurally: upstream batches all 16 experts into one stacked tensor and fuses fc1 into w13, where this checkpoint stores them per-expert and unfused. 3,282 model parameters have no counterpart in the checkpoint. **The fix is a re-quantization from the current Zyphra/ZAYA1-8B**, which produces the modern layout natively. That work is prepared but not yet run, and a re-quantized checkpoint would be a **new artifact** — the accuracy and throughput figures on this card were measured on *this* artifact and would not carry over without being re-measured. Full analysis: [RESEARCH.md` §5.24](https://github.com/t-timms/zaya1-nvfp4-w4a4/blob/master/RESEARCH.md). Everything below this box was accurate on the stack it was measured on and is left unchanged.

*4-bit weights and 4-bit activations* for Zyphra's ZAYA1-8B, on native CUTLASS FP4 tensor-core kernels, built and served inside a 16 GB consumer Blackwell budget (RTX 5070 Ti, SM120). Vendor-official 4-bit releases — Google's Gemma 4 qat-w4a16, Zyphra's own ZAYA1-8B-MXFP4-Experts — are weights-only. These quantize activations too.

Which one do I download?

The 384 BF16 exemptions in this build cost 3.44 GB and buy 0.71 pp of HellaSwag accuracy (95% CI [−1.26, −0.15], paired exact-binomial McNemar over 14,319 items). On a 16 GB card that is usually the wrong trade — start with [`-uniform`](https://huggingface.co/Ttimms/zaya1-8b-nvfp4-w4a4-uniform) and come back here only if you need the extra accuracy or the control artifact.

Before you download

vllm serve Ttimms/zaya1-8b-nvfp4-w4a4 --dtype bfloat16 --enforce-eager \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 5, "prompt_lookup_min": 2}'

Measured, `enforce_eager=True`, median of 5 process invocations: 9.51 tok/s single-stream, 74.4 tok/s at batch-8 (7.82×, 98% of ideal linear scaling). An earlier 102.6 / 407.4 tok/s figure was measured under CUDA graphs and is retracted — see the known-issue section.

Architecture

graph TD
    Base["Zyphra/ZAYA1-8B-legacy<br/>80-layer MoE - reasoning - Apache-2.0"]

    subgraph Build ["Quantization - RTX 5070 Ti, SM120"]
        Cal["llm-compressor NVFP4 W4A4 calibration<br/>4-bit weights + 4-bit activations"]
        Fork{"keep outlier-sensitive<br/>Linears at BF16?"}
    end

    subgraph HF ["Published checkpoints"]
        Mixed["zaya1-8b-nvfp4-w4a4 - 9.46 GB<br/>936 W4A4 + 384 BF16-exempt - eval control"]
        Uni["zaya1-8b-nvfp4-w4a4-uniform - 6.02 GB<br/>1,320 W4A4 - 0 exemptions"]
    end

    subgraph Serve ["Serving & evaluation"]
        vLLM["vLLM - native CUTLASS FP4 tensor-core kernels<br/>enforce_eager=True (CUDA graphs corrupt on SM120)"]
        Bench["HellaSwag acc 46.5% / 45.8%<br/>9.5 tok/s single - ~74 tok/s batch-8"]
    end

    Base --> Cal --> Fork
    Fork -- yes (384) --> Mixed
    Fork -- no --> Uni
    Mixed --> vLLM
    Uni --> vLLM
    vLLM --> Bench

Why there are two checkpoints

`Ttimms/zaya1-8b-nvfp4-w4a4-uniform` is the same model with zero BF16 exemptions — all 1,320 Linears in packed NVFP4 W4A4, at 6.02 GB instead of 9.46 GB, a 36% reduction. This checkpoint keeps 384 outlier-sensitive Linears at BF16, which is where its extra 3.44 GB and its extra 0.71 pp of HellaSwag accuracy both come from. The full paired table is on the -uniform card.

This checkpoint is also the control for that evaluation, so it is kept unchanged.

Manifest metadata note. quantization_manifest.json in this repo records "model": "Zyphra/ZAYA1-8B". The actual base is the pre-refactor 80-layer config, now published as Zyphra/ZAYA1-8B-legacy (see below). The manifest field predates Zyphra's June 2026 refactor and is left as-is rather than rewritten, because this artifact is a published evaluation control.

⚠️ Known issue: CUDA graph capture corrupts output on SM120

The 102.6 / 407.4 tok/s figures previously published here were measured with CUDA graphs enabled — a code path confirmed 2026-08-14 to produce numerically wrong output on this card, independent of MoE backend. A sweep of flashinfer_cutlass (default), cutlass, and marlin (weight-only) all produced garbage under graph capture; only enforce_eager=True generated correctly. Since Marlin barely touches the FP4 MoE path and still failed, the fault is graph capture itself, not any one kernel. Two adjacent upstream issues exist — CUTLASS #3096 (different, non-graph-capture bug) and FlashInfer #2776 (graph-capture-specific, but its stated root cause doesn't explain Marlin failing too) — neither currently offers a fix that preserves CUDA graphs for this failure mode. See Pape, Evertz & Schönherr (arXiv:2605.19537) for the general phenomenon of backend-dependent correctness drift in LLM serving. Full sweep, coherence re-verification, and citations in RESEARCH.md and ROADMAP.md on the GitHub repo.

Set `--enforce-eager` (or `enforce_eager=True`) when serving this checkpoint. It is the only configuration confirmed to produce coherent output.

Throughput and memory (measured, enforce_eager=True)

Measured with vLLM's own benchmark CLI, not a bespoke harness. 5 separate process invocations per configuration, GPU otherwise idle:

vllm bench latency --model <checkpoint> --dtype bfloat16 --kv-cache-dtype fp8 \
  --gpu-memory-utilization <frac> --max-model-len 4096 --no-enable-prefix-caching \
  --input-len 128 --output-len 256 --batch-size <1|8> --enforce-eager \
  --num-iters-warmup 1 --num-iters 1

Throughput is still unchanged between checkpoints — the two land within run-to-run noise of each other at both batch sizes. That part of the original claim held; only the absolute numbers were wrong.

Batching is near-ideal (96–98% of theoretical linear scaling on batch-8), which is evidence the MoE decode step's per-step cost does not grow meaningfully with batch size.

The two checkpoints cannot be run at the same memory fraction on a 16 GB card. This checkpoint fails at --gpu-memory-utilization 0.85 with ValueError: No available memory for the cache blocks, and at 1.0 with Free memory on device cuda:0 (14.66/15.92 GiB) ... less than desired because a desktop session holds ~1.3 GiB. Its working range is narrow; the -uniform build runs at 0.85 with headroom — the practical case for the smaller checkpoint, more so than the accuracy difference.

Retracted, do not cite: 102.6 tok/s single-stream / 407.4 tok/s batch-8 (CUDA graphs; coherent output was never verified at that speed).

An unverified external signal exists and disagrees with the table above. llama.cpp PR #23112's own author reports 45.9 tok/s on a slower RTX 4070 Ti (Q4KM GGUF) — beating this checkpoint's 9.5 tok/s by ~4.8×. Five attempts to reproduce it on this project's own SM120 hardware hit the same non-deterministic hang each time, ruling out the model and toolkit version as causes; looks like a WSL2/driver-level issue. Not resolved as of 2026-08-14 — see GitHub repo RESEARCH.md §5.16 for the full diagnostic log. Update, same day: the gap likely has a principled cause independent of the hang above. Activation quantization (W4A4) gives no speed benefit at batch-1 — decode there is memory-bandwidth-bound, and quantizing activations only helps when compute is the bottleneck. Weight-only quantization is expected to win at batch-1 by design; this checkpoint's advantage is memory footprint and batched throughput (see the batch-8 row above). Detail: RESEARCH.md §5.17. A real batch-1 lever that does exist: vLLM's built-in n-gram speculative decoding gives a validated 2.2× speedup on coding-edit prompts (zero training, one config flag) — no gain on free-form generation, as expected. Deployed to the production serve script and validated live through the real OpenAI-compatible API. Detail: RESEARCH.md §5.18.

This or one of the GGUF builds? Read this first

The most-downloaded ZAYA1-8B quantizations are GGUF k-quant ladders (`Abiray/ZAYA1-8B-GGUF`, `JusteLeo/ZAYA1-8B-GGUF`). Those are weights-only; this is W4A4 — weights and activations. Pick on how you serve, not on bit-width:

Use a GGUF build if you run one conversation at a time, are not on Blackwell, or want a size ladder to fit a specific VRAM budget. Use these if you serve with vLLM on a 50-series card, want throughput under concurrency, or need a checkpoint whose accuracy has actually been measured.

On single-stream speed: the GGUF builds are probably faster, and that is expected

Neither GGUF repo publishes a throughput number, so no measured head-to-head exists. The only figure in circulation comes from PR #23112's own author: 45.9 tok/s on an RTX 4070 Ti with Q4KM — a slower card than the RTX 5070 Ti used here, at roughly 4.8× the 9.5 tok/s measured on this checkpoint. Five attempts to reproduce it on this project's hardware hit the same non-deterministic hang (WSL2/driver-level, not model-level), so treat it as unverified — but it is not disputed here.

There is a principled reason to expect it to hold — though not the usual one. Batch-1 decode is not compute-bound, so quantizing activations has no bottleneck to relieve; W4A4's return arrives under batching, which is where the batch-8 figure above comes from.

But it is not bandwidth-bound either, and that part is worth stating precisely. ~760 M active params at ~4.5 effective bits (NVFP4 + FP8 block scales) is ~0.5 GB of weight traffic per token; at this card's 896 GB/s that is a ~0.56 ms floor — about 1,790 tok/s. The measured 9.5 tok/s is 105 ms/token, or ~0.5% of that roofline. The GPU is idle for essentially the whole decode step, so the real limit is per-step dispatch overhead across 80 sequential layers under the mandatory --enforce-eager, not the weight format. The reported llama.cpp figure sits at ~4.6% of its own roofline — also overhead-bound, just ~9× less so, which explains the gap better than bit-width does.

So a weight-only quant winning at batch-1 is expected, but how much of the gap is the scheme and how much is the missing CUDA graphs is an open question, not a settled one. scripts/sweep_cudagraph_modes.sh in the GitHub repo tests the five vLLM graph modes, with a coherence gate so that no mode can report a throughput number for output nobody verified. On footprint, the fair size-matched comparison to Q5KM (6.43 GB) is the `-uniform` build at 6.02 GB rather than the 9.46 GB mixed build.

If single-stream latency is what you care about, take a GGUF build — and note that `enable_thinking=False` will do far more for your latency than any quantization choice, at a cost this card quantifies and the GGUF cards do not.

Prior art and scope of claims

`switzerchees/ZAYA1-8B-NVFP4` (2026-05-19) is a genuine NVFP4 W4A4 ZAYA1-8B built with NVIDIA ModelOpt, and it predates this checkpoint by two months. Its manifest reports gpu_capability: [12, 0] — the same SM120 compute capability as an RTX 5070 Ti — so SM120 support is not a distinction of this work, and this is not the first W4A4 ZAYA1 checkpoint.

What is distinct, and all that is claimed:

Base model revision — read before reproducing

This checkpoint was quantized from the original 80-layer ZAYA1-8B config (num_hidden_layers: 80, moe_router_topk, zaya_use_eda / zaya_use_mod, transformers 4.57.1).

In late June 2026 Zyphra refactored ZAYA1-8B into upstream-transformers form. Zyphra/ZAYA1-8B now reports num_hidden_layers: 40 with layer_types: hybrid and num_experts_per_tok, and the original was moved to `Zyphra/ZAYA1-8B-legacy`.

Core dimensions are unchanged (hidden 2048, 16 experts, top-1 routing, vocab 262272), so this reads as a re-expression of the same model rather than a new one — but reproduce against `Zyphra/ZAYA1-8B-legacy`, or you will hit an architecture mismatch.

Verified, not just inferred (2026-08-14): fetched model.embed_tokens.weight directly from both repos via HTTP range request (no full download) and compared the raw bytes — byte-for-byte identical, 1,074,266,112 bytes, BF16, shape [262272, 2048]. The two repos' safetensors index files also report an identical aggregate total_size (17,680,978,928 bytes) despite the tensor layout changing from 2,483 named tensors (legacy) to 1,283 (current, fused/batched expert tensors, transformers-conventional naming). This is why base_model lists both repos below with base_model_relation: quantized — not an assumption, a direct measurement on the one tensor that could be checked without reverse-engineering the expert-fusion mapping. Every other published NVFP4/GGUF/BNB quantization of this model tags only the current Zyphra/ZAYA1-8B — this checkpoint is the only one built from and verified against the pre-refactor structure directly.

Highlights

Quantization details

See quantization_manifest.json for the full machine-readable config, including the exact outlier-layer list and mixed-precision threshold.

Why this is hard

ZAYA1-8B is an 80-layer MoE (760M active / 8.4B total) with Zyphra's CCA (compressed convolutional attention) — no stock quantization path works out of the box. W4A4 requires calibrating activation scales, not just weights; the compressed-tensors calibration path has a silent NaN-producing trap if you calibrate through the fake-quant nn.Linear.forward wrapper; and the NVFP4 global-scale convention (2688 / max_abs, divisor form, block scales pre-multiplied) is undocumented — getting it wrong produces silent pad-token collapse, not an error. Full root-cause writeup in RESEARCH.md on the GitHub repo.

Usage

Requires vLLM built from source with SM120 NVFP4 CUTLASS kernels (stock wheels don't include them) — see the reproduce steps. Inference must run in bfloat16 (not fp16/fp32).

# vLLM built from source with: TORCH_CUDA_ARCH_LIST=12.0
vllm serve Ttimms/zaya1-8b-nvfp4-w4a4 --dtype bfloat16 --enforce-eager \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 5, "prompt_lookup_min": 2}'

`--enforce-eager` is required for correct output, not just recommended — CUDA graph capture corrupts generation on this card regardless of MoE backend. See "Known issue" above before serving without it.

`--speculative-config` is optional but free — lossless n-gram speculative decoding, validated 2.2× faster on coding-edit prompts (no gain on free-form generation, since there's no prompt/output overlap to exploit). Detail: RESEARCH.md §5.18. The exact command above is `scripts/serve.sh` in the repo.

Evaluation

Generative benchmarks were measured on the companion 6.02 GB uniform checkpoint (`Ttimms/zaya1-8b-nvfp4-w4a4-uniform`), not this one: HumanEval 72.6% pass@1 (95% CI [65.3, 78.8]), GSM8K 65.5% [62.9, 68.0], MMLU-Pro 0-shot 48.1% [44.5, 51.8]. The two checkpoints differ by −0.71 pp HellaSwag on a paired 14,319-item test, so those figures are indicative here but were not measured on this checkpoint. ⚡ `enable_thinking=False` is 8.5× faster but costs 17–29 accuracy points (HumanEval −28.66, MMLU-Pro −21.43, GSM8K −17.36; all p<0.0001, paired McNemar). *ZAYA1's accuracy is its reasoning, and its reasoning is what makes it slow — they cannot be separated. Use the flag for per-request routing (`chat_template_kwargs`), not as a global switch. If you need low latency more than accuracy, a weight-only quant of a non-reasoning model will serve you better. MMLU-Pro is 0-shot and is not comparable to Zyphra's 5-shot 74.2%* — that gap is a protocol difference, not quantization damage. Full analysis: `RESEARCH.md` §5.22.

Budget-forced GPQA-Diamond using an s1-style harness that caps the reasoning trace and scores only the closed answer — stock lm-eval harnesses score near random on this model because ZAYA never closes its <think> block within a normal budget and answers in \boxed{} format:

Sample size caveat. n=24 (paired). The 95% binomial confidence interval at 62.5% is roughly 41–81% — about 40 points wide. This is not evidence of parity with Zyphra's BF16 CoT figure of 71.0%; the interval is too wide to distinguish most hypotheses. What the data does support is the monotonic rise with reasoning budget, which is a checkpoint-health signal: a damaged checkpoint would not improve with more think tokens. A higher-n run is the most valuable open item on this checkpoint.

The gap to BF16 is consistent with the 16 GB context/reasoning-budget ceiling on the source hardware rather than quantization damage, but at n=24 that remains an interpretation, not a measurement.

License

Apache 2.0 — matches the ZAYA1-8B upstream license.