Datasetpublic

Relativ3pa1n/dsv4-flash-sm86-8x3090

DeepSeek-V4-Flash on 8x RTX 3090 (SM86): 262K context, 120 tok/s aggregate Serving recipes, launch wrappers, and the measured throughput/context ladder for running a W4A16 DeepSeek-V4-Flash-class model on 8x RTX 3090 (SM 8.6, 24 GiB each) with CUDA graph decode, FlashInfer sparse MLA, Marlin MoE, and compressed hybrid KV. The ladder Concurrent sequences amortize the TP8 allreduce that dominates each decode step, so aggregate throughput scales near-linear while… See the full description on the dataset page: https://huggingface.co/datasets/Relativ3pa1n/dsv4-flash-sm86-8x3090.

sourceHugging Faceotherupdated 1mo agoView on Hugging Face

0likes36downloads

Dataset Card

DeepSeek-V4-Flash on 8x RTX 3090 (SM86): 262K context, 120 tok/s aggregate

Serving recipes, launch wrappers, and the measured throughput/context ladder for running a W4A16 DeepSeek-V4-Flash-class model on 8x RTX 3090 (SM 8.6, 24 GiB each) with CUDA graph decode, FlashInfer sparse MLA, Marlin MoE, and compressed hybrid KV.

The ladder

Concurrent sequences amortize the TP8 allreduce that dominates each decode step, so aggregate throughput scales near-linear while per-sequence context trades down.

Envelope	Aggregate decode	Per-sequence
1 x 262K	59.6 tok/s	59.6
2 x 176K	81.3 tok/s	39.3 / 42.0
3 x 114K	104.5 tok/s	32.7 / 36.0 / 35.8
4 x 80K	120.1 tok/s	29.2 / 30.3 / 30.3 / 30.2

All rows coherent, verified with exact-count needle probes at full context. Prefill at the 262K frontier: ~451 tok/s. 258K-token needle pass.

Model

Intended public checkpoint for reproduction: Intel/DeepSeek-V4-Flash-W4A16-AutoRound (AutoRound W4A16, group size 128). Quantization is an input checkpoint property; the SM86 work in this repo is the serving stack. The full build chain for an abliterated variant is in docs/MODEL_PREP.md, and the verbatim quantization config ships in reference/config.json.

—docs/MODEL_PREP.md: the checkpoint build chain: official FP8 base, rank-1 abliteration (pinned tooling, direction file, parameters), and the deterministic model-free AutoRound W4A16 recipe. Every input is public and pinned by revision and hash.
—docs/SM86_DSV4_RUNBOOK.md: hardware/software pins, reproduced results, launch commands, memory knobs, verification checklist, and the rank0 trace breakdown (17.5 of 18 ms per graph step is TP8 allreduce).
—docs/EXPLOIT_BENCH.md: how far up the V8 exploitation ladder the checkpoint climbs (ExploitBench v8-bench, three patched CVEs, deterministic grader). Full T5 coverage on every env, one differential signal, zero exploit primitives.
—docs/TERMINAL_BENCH.md: a coherence check for the served checkpoint: real agentic terminal work under an unseen harness, with a same-rig calibration baseline. 5/18 solved, multi-step tool loops intact.
—docs/SM86_256K_ROADMAP.md: the context/sequence ladder methodology, what to keep and avoid, NCCL matrix, profile buckets, and kernel-work order after profiling.
—scripts/: the validated launch wrappers, verbatim. The common launcher prints the full argv, git revision, import origin, and memory knobs at startup; debug from that manifest.

Minimum reproduction

bash

scripts/serve_sm86_262k_cudagraph.sh

Maximum throughput:

bash

scripts/serve_sm86_80k_4seq_cudagraph.sh

Software pins

Python 3.12.12, PyTorch 2.11.0+cu130, CUDA 13.0.1 rebuild with TORCH_CUDA_ARCH_LIST=8.6, FlashInfer 0.6.14+sm89.1 (JIT for SM86), Marlin MoE, kv_cache_dtype=fp8_ds_mla, attention_backend=FLASHINFER_MLA_SPARSE_DSV4. Full details in the runbook.

How this was made

This stack was produced by an orchestrated agent workflow rather than a human engineering team: PAI (a personal AI infrastructure managing the rig, boots, benches, and traces) directed a GPT-5.5 coding agent over roughly three weeks of iterative kernel and backend work. The coding agent authored the technical direction, patches, and launch envelopes; PAI executed every command verbatim and verified each result before it was banked. Every number in this repository was reproduced through that loop.

Orchestration mechanics

Everything ran through one tmux server on the orchestrator laptop. Each tmux pane held a persistent SSH session to one machine, and PAI drove them by typing into panes (tmux send-keys) and reading the screen back (capture-pane): no agent SDKs, no APIs, a terminal multiplexer as the universal interface.

Box	Role
Orchestrator laptop	PAI, the tmux server, every SSH session
machineA (8x3090 rig)	GPU serving, boots, benches, traces
machineB (workstation)	GPT-5.5 coding-agent sandbox, campaign agents
local-git (self-hosted Gitea)	file bus between boxes

The coding agent's sandbox had no outbound internet, so all code, patches, and docs moved through the local Gitea: the agent pushed, the orchestrator pulled on the rig and ran verbatim, and results flowed back the same way. Because the SSH sessions lived inside tmux, multi-hour GPU boots survived laptop disconnects, and the orchestrator re-read pane scrollback to verify what actually happened before banking any number.

Sources

—DeepSeek-V4 technical report: https://arxiv.org/html/2606.19348v1
—CSA/HCA walkthrough: https://prathamp.com/blog/deepseek-v4-csa-and-hca/
—Ampere community fork: https://github.com/Lasimeri/vllm-dsv4-ampere

License

Launch wrappers and docs follow the repository license. Model weights follow their upstream licenses.