Modelpublic

0xTank/DeepSeek-V4.1-Flash-vLLM-4x-GB10-Recipe

sourceHugging Faceotherupdated 13d agoView on Hugging Face

1likes

Model Card

DeepSeek-V4.1-Flash — 4× GB10, vLLM, K3, 600K recipe

A deployment recipe using the original DeepSeek model weights. Published by 0xTank for four 128-GB NVIDIA GB10 systems using TP4 over RoCE. The working configuration boots with a 600,000-token context ceiling, DSpark K3, and FULL_AND_PIECEWISE CUDA graphs. This repository includes the required custom Python files, upstream patches, parameterized launcher, pinned build instructions, Engram staging tool, benchmark scripts, and raw benchmark results.

Credit first: DeepSeek created the model and checkpoint, including its draft layers. The foundational four-node Spark recipe and its disk-backed Engram, staging, page-layout, and SM12x indexer work come from Tech2Wild/Kai (tonyd2wild). vLLM, NVIDIA, FlashInfer, Triton, PyTorch and their contributors provide the underlying inference, communication, and kernel stack. See NOTICE.md.

What 0xTank changed

1.Graph startup-state fix: skip the throwaway graph-memory profiling pass and clear KV/Engram startup state after capture, in place. The buffer addresses used by graphs remain intact. The combined startup fix was validated on the serving fleet.
2.Compact output-projection decode kernel: retain the original MXFP8 projection values alongside the existing BF16 fallback and use an opt-in Triton reduction for supported small decode shapes. The guard accepts 1–8 token rows per forward, not 1–8 users. Larger/unsupported shapes use the BF16 fallback; this includes prefill. The original checkpoint is used unchanged.
3.K3 operating configuration: use three speculative tokens with adaptive verification off and matching capture sizes.
4.Pinned runtime build and reproducibility: rebuild the stable vLLM extension, pin FlashInfer/submodules, prebuild key kernels, and limit build parallelism, extending the upstream build approach.
5.600K ceiling and operational packaging: validate boot and short-prompt generation at 600K; parameterize node addresses/paths. The public launcher refuses to delete existing containers and flushes only relevant filesystems before cache dropping, avoiding unrelated USB backup waits.

The exact four 0xTank patch files are in patches/0xTank/; the seven inherited runtime patches are separately kept in patches/upstream/. Their source headers and license notices are retained. All release files have SHA-256 checksums.

Results — fresh 600K/K3 prose run

Measured on 2026-09-11 on the running four-GB10 configuration: 946 prompt tokens and 512 generated tokens per request, temperature 0, thinking off, one request at a time. One fresh prompt was followed by five identical warm repeats. Cold here means a fresh prompt on the loaded server.

Prose condition	Decode tokens/sec	TTFT seconds
Cold — fresh prompt	36.13	0.733
Warm — median of five repeats	36.67	0.385
Warm — observed range	34.40–37.22	0.383–0.432

All six requests completed 512 generated tokens and passed the prose sanity checks. Request-count telemetry observed one running request at most and zero waiting requests. The configured context ceiling is 600,000 tokens; the measurements above use the 946-token prose prompt.

See BENCHMARKS.md for every request and the timing formula. The complete fresh-run records, prompt, generated text and traffic samples are in prose-600k-fresh.jsonl.

Full Tech2Wild/Kai benchmark

The complete fixed-prompt performance suite covers coding, JSON, narrative, prose, math, reasoning, summarization and table formatting at concurrency 1–6, plus counting and four prefill sizes. The separate needle-retrieval test uses its default 131,072-token target and 50% needle depth.

See FULL-BENCHMARK.md for this recipe's complete results, test conditions, raw records and reproduction commands. Workloads and timing definitions come from Tony's pinned benchmark; 0xTank adds raw-response capture. This is a standalone measurement of the 600K/K3 recipe.

Requirements and preparation

—Four GB10 / SM 12.1 machines, 128 GB unified memory each; tested Linux kernel 6.17.0-1026-nvidia, NVIDIA driver 580.159.03, PyTorch 2.13.0+cu130.
—Docker with NVIDIA GPU runtime and working RoCE/RDMA on all four nodes. Interface/HCA/GID/subnet values must match your hardware. Do not assume the example network is correct for your system.
—Approximately 510.3 GB / 475.25 GiB for the original model; sufficient additional NVMe space for runtime build/image/cache and each worker's sparse Engram copy (roughly 48 GB allocated). Sparse-file-aware storage is required for these copies.
—The head reads the model from local NVMe. Workers need the full checkpoint visible through a read-only NFS mount, plus their own node-local Engram rows.
—The launcher requires at least 100 GiB available host memory before loading. It uses the measured 112-GiB container memory/swap limit. On GB10, container memory counters do not account for all GPU/unified-memory use.
—The API is not authenticated by this example. Keep port 8000 and distributed ports on a trusted network or behind an authenticated gateway. This is not an internet-facing security configuration.

Download the weights

The exact original weights are available directly from the 0xTank weight mirror. DeepSeek created the model; the weights are unchanged. The complete pinned snapshot has 88 files totaling 510,313,353,565 bytes (about 510.3 GB). Its weight hashes and original file metadata match the DeepSeek source.

Download on the head (requires huggingface_hub):

bash

hf download 0xTank/DeepSeek-V4.1-Flash \
  --revision 39a872b296d3570e4a5bbb27138bbfa1a9755e50 \
  --local-dir /srv/models/DeepSeek-V4.1-Flash

Configure read-only NFS export/mounts so MODEL_PATH resolves to that model directory on every node. Consult the upstream full recipe for the original network/storage setup. Do not run its old launch commands on top of this deployment; this package supplies the K3/600K launcher.

Build the runtime

Run on a GB10 build machine with ample free memory and disk, not alongside an active model. Clone/download this recipe first, then:

bash

bash build/build.sh /absolute/path/to/new-vllm-build

The build uses the pinned vLLM source and base image, the stable-only extension build, and the pinned FlashInfer build from this deployment. It creates vllm-dsv41:overlay5. Build on each node or use docker image save / docker image load to distribute your built image to the other three nodes. No private container snapshots, binary runtime archives, credentials, or model weights are included in this public repository.

Validation: the underlying build steps and patch stack ran on the test fleet. The public launcher passed syntax and four-rank dry-run checks. This package provides the pinned source stack; compiled and tuning caches are generated locally during build/startup. See PROVENANCE.md for revisions.

On each worker, create its Engram copy using its rank (1, 2, or 3):

bash

bash tools/prepare-engram.sh 1 /srv/models/DeepSeek-V4.1-Flash \
  /srv/engram-local/DeepSeek-V4.1-Flash

Use the matching rank on each worker. The ranges are specific to the pinned checkpoint and TP4; do not reuse them for another model/revision. The tool checks sampled rows against the source. The head does not need this copy.

Launch and verify

Copy the entire recipe to each node. Copy launch/cluster.env.example to cluster.env and edit addresses, paths, interface, HCA and GID. Each node may use different local paths, but model/settings must match. Inspect the generated command without starting anything:

bash

bash launch/node.sh 0 ./cluster.env --dry-run

Stop any old deployment head first, then workers; preserve its containers, images, patches and caches. This launcher deliberately refuses an existing container name. Start the new ranks on their respective nodes in this order:

bash

bash launch/node.sh 3 ./cluster.env  # on rank 3
bash launch/node.sh 2 ./cluster.env  # on rank 2
bash launch/node.sh 1 ./cluster.env  # on rank 1
bash launch/node.sh 0 ./cluster.env  # on the head

All four must be started promptly. Weight loading, draft loading, autotuning and graph capture can take many minutes; a container being running is not proof that the API is ready. Inspect logs before restarting. Expect retained compact projections (43 per rank in our run), graph-capture completion, and startup cleanup of 54 KV tensors per rank, followed by API startup.

On the head, after /health responds successfully:

bash

curl --fail http://127.0.0.1:8000/health
curl --fail http://127.0.0.1:8000/v1/models
python3 bench/prose-six.py

The prose test sends one fresh prompt followed by five warm repeats, sequentially, with 512 generated tokens per request. Avoid other API traffic while measuring. Thinking is off by default. Vision (up to four images) and tool parsers are enabled; tools/vision_tools_demo.py provides additional endpoint checks.