Modelpublic

Baekpica/DeepSeek-V4-Flash-0731-120B-REAM-104E-NVFP4

sourceHugging Facemitupdated 29d agoView on Hugging Face

4likes1.3kdownloads

Model Card

DeepSeek-V4-Flash-0731 120B REAM 104E NVFP4

Support my work

I work on making large language models practical on hardware they were never really designed to fit on — through mixed quantization, inference optimization, custom kernels, and serving experiments.

While much of the development happens on local hardware, calibration, profiling, and large-scale validation often require expensive on-demand GPUs.

Contributions help pay for that compute, storage, and testing infrastructure, so I can keep experimenting and publishing the results openly.

This is the predominantly weight-only NVFP4A16 version of `Baekpica/DeepSeek-V4-Flash-0731-120B-REAM-104E-BF16`.

—Routed-expert and other eligible weights: NVIDIA FP4 E2M1, group size 16, FP8 group scales plus a global scale.
—Activations are unquantized 16-bit (A16) on the NVFP4 paths; this is not the W4A4 NVFP4 preset. The grouped attention WO_A compatibility path is the explicit mixed-precision exception described below.
—Quantizer: vLLM Project llm-compressor model-free PTQ, scheme="NVFP4A16", using data-free RTN. The shorter repository suffix is NVFP4; the checkpoint is still weight-only W4A16, not W4A4.
—MoE router modules, embeddings, normalization layers, and the output head remain unquantized. The 145 small attention-compressor/indexer score projections also remain BF16 because native DeepSeek-V4 serving instantiates those kernels with quant_config=None. The 43 grouped WO_A attention projections use dynamic W8A8 E4M3 block FP8 (128×128 weight blocks, 128-element activation groups) because the native CUDA inverse-RoPE output einsum directly consumes that grouped FP8 layout. Linear gate_proj weights, including shared-expert projections, remain part of the NVFP4 fused gate/up quantization path.
—The checkpoint has eight safetensors shards. The largest is 9,399,293,182 bytes, below the decimal 10 GB cap, to reduce per-file load peaks on constrained nodes such as DGX Spark.
—Intended hardware: NVIDIA Blackwell (SM100) or newer with an inference stack supporting compressed-tensors nvfp4-pack-quantized checkpoints.

The parent model has 119,821,633,111 logical main-model parameters, 43 decoder layers, 104 routed experts per layer, top-6 routing, and approximately 13.802B active parameters/token. DSpark/MTP auxiliary weights are not included.

Build provenance is recorded in nvfp4a16_provenance.json; the checkpoint header/config audit is in checkpoint_audit.json. See the BF16 parent model card for the direct-saliency REAM and calibration details.

Loading

vLLM 0.26.0 compatibility fork

The single-GPU configuration below was verified on NVIDIA GB10 using the `Baekpica/vllm` branch `deepseek-v4-v0.26.0`, based on the upstream v0.26.0 tag. The compatibility changes are pinned in commit `622008c4e`:

diff

# Grouped FP8 WO_A uses the DeepGEMM scale attribute after loading.
- weight_scale
+ weight_scale_inv

# The fused CUDA router has no 104-expert instantiation.
- if current_platform.is_xpu():
+ if current_platform.is_xpu() or gating_output.shape[-1] == 104:

Install the fork with the precompiled vLLM extension, then install the matching FlashInfer packages. The CUDA-specific index is required for the JIT-cache wheel used by the validated CUDA 13.0 environment.

bash

git clone --branch deepseek-v4-v0.26.0 --single-branch \\
  https://github.com/Baekpica/vllm.git
cd vllm

uv venv --python 3.12
source .venv/bin/activate
VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto

uv pip install \\
  flashinfer-python==0.6.14 \\
  flashinfer-cubin==0.6.14 \\
  'flashinfer-jit-cache==0.6.14+cu130' \\
  --extra-index-url https://flashinfer.ai/whl \\
  --extra-index-url https://flashinfer.ai/whl/cu130

Use an nvcc/header pair that matches the PyTorch CUDA build. The validated DGX Spark environment used /usr/local/cuda-13.0:

bash

export CUDA_HOME=/usr/local/cuda-13.0
export PATH="$CUDA_HOME/bin:$PATH"

vllm serve Baekpica/DeepSeek-V4-Flash-0731-120B-REAM-104E-NVFP4 \\
  --port 8001 \\
  --optimization-level 3 \\
  --max-model-len 196608 \\
  --max-num-seqs 2 \\
  --max-num-batched-tokens 16384 \\
  --gpu-memory-utilization 0.92 \\
  --enable-prefix-caching \\
  --enable-chunked-prefill \\
  --kv-cache-dtype fp8 \\
  --moe-backend auto \\
  --attention-backend flashinfer \\
  --tokenizer-mode deepseek_v4 \\
  --reasoning-parser deepseek_v4 \\
  --tool-call-parser deepseek_v4 \\
  --enable-auto-tool-choice \\
  --quantization compressed-tensors \\
  --language-model-only

The checkpoint quantization_config declares the packed format and ignore list. No runtime adapter or model-repository custom code is required.

The unquantized output tensor intentionally retains the original DeepSeek-V4 checkpoint name head.weight; vLLM's native DeepSeek-V4 mapper exposes it as lm_head.weight at runtime. Tensor paths otherwise use the original native DeepSeek-V4 layout expected by that mapper; no adapter or custom model file is needed.

vLLM serving validation

Single NVIDIA GB10 (vLLM 0.26.0)

The compatibility branch and 196,608-token launch configuration above were verified to load and serve this checkpoint on DGX Spark / NVIDIA GB10.

4× NVIDIA B200 (vLLM 0.22.0)

Validated on 4× NVIDIA B200 with vLLM 0.22.0, tensor parallel 4, expert parallel enabled, 8,192-token context, FP8 KV cache, and CUDA graphs enabled. The server completed weight loading, the 104-expert routing warmup, CUDA graph capture, /health (HTTP 200), model discovery, and four OpenAI-compatible chat requests. Raw responses and timings are in vllm_serving_smoke_2048.json.

Case	Result	Completion
Korean Python	Format rubric 4/4; semantic review found incorrect edge-case assertions	490 tokens, `stop`
Reservoir math	Failed; arithmetic error followed by repetition	2,048 tokens, `length`
5,810-token needle	Exact retrieval	55 tokens, `stop`
Strict JSON	Exact output	29 tokens, `stop`

This smoke test demonstrates that the standalone repository serves end to end; it is not evidence of benchmark-level quality. In particular, the Korean code was fluent and structurally complete but should not be treated as a correctness pass because some generated test expectations were inconsistent with the problem definition.

Limitations

This is RTN weight-only quantization of an expert-merged model. Quantization and expert merging can both affect quality. The 2,048-token serving smoke exposed a basic arithmetic failure/repetition and incorrect coding edge-case assertions, while long-context retrieval and strict JSON succeeded. Korean fluency and surface code structure therefore do not guarantee logical correctness. Run task-specific evaluations and sandbox generated code before production use. The source model's MIT license and usage considerations apply.