CoolFace
Modelpublic

anemll/DeepSeek-V4-Flash-NVFP4-TP2-W4A4-v1

sourceHugging Facemitupdated 2mo agoView on Hugging Face
3likes154downloads
Model Card

DeepSeek-V4-Flash NVFP4 TP2 W4A4 v1

This is a single-download, two-node GB10 serving bundle. It combines an offline-prepared NVIDIA NVFP4 W4A4 target with the native three-stage DSpark speculative draft.

This is not a drop-in Transformers checkpoint. The target has a custom rank-contiguous TP=2 physical layout and requires the matching loader in Anemll/dspark-vllm-gx10, branch `dspark-nvfp4-a4w4`.

Bundle layout

PathContentsRuntime role
repository rootPrepared NVIDIA NVFP4 targetW4A4 FlashInfer CUTLASS target
dspark/Native mtp.0, mtp.1, and mtp.2 shards plus a draft-only indexThree-stage DSpark draft

The root target contains 87 Safetensors files, 3,483 tensors, and 168,266,881,608 payload bytes. The draft contains 4,705 tensors and 10,862,838,300 payload bytes. bundle-manifest.json records the complete layout and provenance.

The target and draft remain two explicit model paths at runtime: mount the repository root as the target and dspark/ as the speculative model.

Requirements

  • —two NVIDIA DGX Spark or equivalent GB10/SM121 nodes;
  • —TP=2 with a dedicated node-to-node fabric;
  • —enough local storage for the complete bundle on each node;
  • —the custom vLLM runtime branch linked above;
  • —an ARM64 runtime image built from that branch.

The checkpoint is not supported as a one-node TP=1 model. The prepared target format itself is pinned to TP=2.

Download

Run on both nodes, choosing a local SSD path with sufficient free space:

bash
hf download anemll/DeepSeek-V4-Flash-NVFP4-TP2-W4A4-v1 \
  --local-dir /srv/dspark/models/DeepSeek-V4-Flash-NVFP4-TP2-W4A4-v1

Runtime setup

Clone the matching runtime branch on both nodes:

bash
git clone --branch dspark-nvfp4-a4w4 \
  https://github.com/Anemll/dspark-vllm-gx10.git
cd dspark-vllm-gx10

Use an image built from this branch. The legacy 0.1.1 image does not contain the prepared bulk direct reader unless it has been rebuilt from the branch:

bash
FINAL_IMAGE=dspark-vllm-gx10:nvfp4-a4w4-v1 ./scripts/build-image.sh

Set the following values in both config/head.env and config/worker.env, in addition to their normal rank, fabric, NCCL, cache, and rendezvous fields:

bash
DSPARK_VLLM_IMAGE=dspark-vllm-gx10:nvfp4-a4w4-v1
DSPARK_MODEL_HOST=/srv/dspark/models/DeepSeek-V4-Flash-NVFP4-TP2-W4A4-v1
DSPARK_DRAFT_MODEL_HOST=/srv/dspark/models/DeepSeek-V4-Flash-NVFP4-TP2-W4A4-v1/dspark
SERVED_MODEL_NAME=deepseek-v4-flash-nvfp4-dspark

DSPARK_WEIGHT_LOAD_FORMAT=auto
DSPARK_MOE_BACKEND=auto
DSPARK_SPECULATION_MODE=dspark
MTP_NUM_TOKENS=5
VLLM_DSPARK_CONFIDENCE_SCHEDULER=off
VLLM_DSPARK_CONFIDENCE_THRESHOLD=0.0

VLLM_DSV4_NVFP4_CUTLASS_PREPARED_LOAD=1
VLLM_DSV4_NVFP4_CUTLASS_PREPARED_MANIFEST_SHA256=REPLACE_WITH_MANIFEST_SHA256
VLLM_DSV4_NVFP4_CUTLASS_PREPARED_DIRECT_READ=1

Use the first field of dspark-nvfp4-tp2-repack.json.sha256 as the manifest value. Keep the global MoE backend at auto: the prepared target is scoped to FlashInfer CUTLASS by its loader, while the native DSpark draft requires its own backend selection. Direct reads are the prepared-loader default; 0 selects the slower diagnostic mmap path.

Start rank 1 first, then rank 0:

bash
# WORKER / rank 1
./scripts/start-node.sh config/worker.env

# HEAD / rank 0, after rank 1 enters the rendezvous
./scripts/start-node.sh config/head.env

When passwordless worker SSH is configured in the head environment, the head can launch both ranks:

bash
./scripts/start-cluster.sh config/head.env config/worker.env

Startup verification

Follow the container log on either rank:

bash
cid="$(sudo docker ps -q --filter label=com.docker.compose.service=vllm-dspark)"
sudo docker logs -f "$cid"

A valid prepared bulk load reports:

  • —NVFP4_PREPARED event=enabled ... io_mode=preadv;
  • —43 event=layer_load records;
  • —event=complete layers=43 reads=344 copies=344 ... io_mode=preadv;
  • —43 post-load records with transforms=0 and backend FLASHINFER_CUTLASS.

After startup:

bash
curl -fsS http://HEAD_HOST:8888/health
curl -fsS http://HEAD_HOST:8888/v1/models

Validated load performance

The bulk direct reader was validated end to end on two GB10 nodes:

MeasurementPrevious prepared pathBulk direct readerImprovement
Slower-rank target load558.19 s65.23 s8.56x
Complete head model load595.90 s108.54 s5.49x
Head start to API servern/a231.64 sn/a

The run reached HTTP health 200, exposed the expected model, kept both ranks out of OOM state, and returned a coherent smoke response. Raw evidence is in `nvfp4-prepared-direct-read-full-3689b1c.json`.

Prefill comparison

These are warmed, server-side prefill rates on the same two-node TP=2 cluster. Prefill is target-only and does not benefit from speculative decoding.

Input tokensFP8/B12X productionNVFP4 W4A4GainProduction TTFTW4A4 TTFT
1,0242,033.0 tok/s2,242.5 tok/s+10.3%0.512 s0.463 s
2,0482,252.0 tok/s2,473.2 tok/s+9.8%0.920 s0.835 s
4,0962,320.7 tok/s2,659.3 tok/s+14.6%1.776 s1.552 s
8,1922,184.2 tok/s2,593.5 tok/s+18.7%3.765 s3.173 s
16,3842,203.8 tok/s2,501.7 tok/s+13.5%7.455 s6.573 s
32,7682,176.1 tok/s2,477.3 tok/s+13.8%15.119 s13.264 s

The reports are same-size aggregates, not paired identical-prompt trials. See the comparison and raw-artifact links.

Decode and agentic comparisons

All rows use MTP=5, confidence scheduling off, probabilistic draft sampling, no draft/verify overlap optimization, temperature zero, and 512 requested tokens per stream.

WorkloadCFP8/B12X + DSparkW4A4 + DSparkComparisonW4A4 mean accepted length
Canonical chat/tool4105.48 tok/s96.02 tok/s-9.0%3.127
tool_agentic8not archived cleanly360.68 tok/sno matched control5.270

The canonical control is a three-trial median; its W4A4 row is one post-promotion run. The agentic row is a clean W4A4 production measurement; the older timing-instrumented concurrency study is deliberately not used as a control. The agentic prompt SHA-256 is 6173a7ae0ea3c64b364d0c405be28808efb8486c68a7011e966d31ce222c1736, and all eight W4A4 streams passed the automated no-collapse diagnostic.

The report is intentionally mixed rather than marketing a single decode number: W4A4 lost about 9% in the exploratory canonical comparison, while the high-acceptance agentic path reached 360.68 aggregate tok/s without a clean legacy control. See the full serving comparison and raw JSON links.

Limitations

  • —This artifact is tied to the custom prepared-loader contract and TP=2.
  • —It is not supported by stock Transformers or stock vLLM.
  • —The target and speculative draft have separate physical layouts and must be mounted at the two paths shown above.
  • —Decode performance depends strongly on prompt content and draft acceptance; report prompt, concurrency, acceptance, output length, and timing convention with every result.
  • —The agentic no-collapse diagnostic is not a comprehensive quality evaluation.
  • —The target originates from NVIDIA's NVFP4 checkpoint and the draft from the native DeepSeek V4 Flash DSpark lineage; review the upstream licenses and terms as well as the files included here.

Reproducibility