Modelpublic

0xTank/Hy4-preview-NVFP4-W4A16-MTP

sourceHugging Faceupdated 10d agoView on Hugging Face

1likes426downloads

Model Card

Hy4-preview NVFP4 W4A16 + native MTP — 200K TP4/DCP4 revision

Runtime revision: 2026-09-08, 200K decode graphs + allocator reclaim. Same model weights; this is a serving-runtime update, not a new quantization or pruned model.

Upload status: complete. All 130 model shards, checkpoint metadata and required gate/indexer overlays are present and remote sizes verified. Newly uploaded files were also SHA-256 verified. Runtime prerequisites and qualification limitations below still apply.

Verified local configuration

Four NVIDIA GB10/DGX Spark systems, TP4 and decode-context parallelism 4 over RoCE, native MTP k=4, all experts retained. Context setting 200,000 tokens; runtime reports 204,859 usable cache tokens. NVFP4 MLA cache, 1.9 GB cache budget per rank, C1, prefill batch 256. Full-decode-only CUDA graphs with sizes 1 and 5.

Weights occupy 104.83 GiB of model tensor memory per rank. The local 500 MiB MemAvailable reserve remains enforced. This is a tight-memory configuration, not suitable for adding concurrent workloads without a new preflight.

Current measured benchmarks

Rendered 3,000-token prompts, temperature 0, thinking off, maximum 256 generated tokens. Cold means no matching cached prefix on an already-loaded model; warm reuses 2,560 prefix tokens. Warm effective prefill is not raw compute throughput.

Prompt	Cache	Prefill tok/s	TTFT seconds	Decode tok/s
Natural prose	Cold	205.0	14.65	14.71
Natural prose	Warm	1,361.2 effective	2.23	14.51
Integer sequence	Cold	231.9	12.95	31.10
Integer sequence	Warm	1,502.2 effective	2.02	31.14

Previous 64K k=4 prose results were 9.66 cold / 9.26 warm tok/s. Decode improved approximately 52% / 57% on these samples. Prefill did not materially improve. Results are prompt-dependent and are not an aggregate multi-session claim.

Both prose responses were coherent explanations of accessibility and public libraries. Counting returned consecutive complete integers, with the final integer truncated by the output limit. A separate 8,192-token retrieval test correctly recovered a code from the beginning; sampled Acer memory stayed above 580 MiB. These are limited checks, not broad quality benchmarks.

Long-context limitation: 200K is configured and allocated, but an end-to-end 200K prompt has not been qualified. The 64K test stopped before generation at its 550 MiB pre-test threshold. Do not treat the 8K retrieval result as validation of 200K quality or stability.

What changed

—Integrated NVFP4 MLA with native four-rank DCP gathering/combination and global sparse-indexer selection.
—Included the DCP communicator in the runtime graph-capture context.
—Enabled full decode graphs and native MTP graph capture.
—Preserved gate/indexer overlays, early MTP sharing and iHC optimizations from the working build.
—Added garbage collection and glibc allocator trimming before KV allocation; reclaimed approximately 0.9 GiB per rank in the successful run.
—Set MALLOC_ARENA_MAX=2; retained the 500 MiB reserve and 768 MiB post-KV runtime allowance.

See recipe/WORKING-RECIPE.md, the launch and patch files, and benchmarks/ for measured evidence. The supplied scripts are a revision of an existing custom image/work directory, not a standalone bootstrap installer. The full pinned image, vendor runtime, overlays and checkpoint are required; do not assume stock vLLM can load this recipe unchanged. Deployment addresses and local paths in published scripts are placeholders to configure for your fleet.

Thinking off uses chat_template_kwargs: {"reasoning_effort":"no_think"} in the request. Preserve the model's upstream license in LICENSE and review it before redistribution or use.