CoolFace
Modelpublic

PiehSoft/DeepSeek-R1-Distill-Llama-70B-NVFP4

sourceHugging Facemitupdated 5mo agoView on Hugging Face
0likes78downloads
Model Card

DeepSeek-R1-Distill-Llama-70B-NVFP4

NVFP4 quantized version of DeepSeek-R1-Distill-Llama-70B using custom Blackwell NVFP4 GEMM kernels.

140 GB → 40 GB (0.29x) with vision tower excluded.

NVFP4 Quantization Details

PropertyValue
Base modeldeepseek-ai/DeepSeek-R1-Distill-Llama-70B
QuantizationNVFP4 (W4A4 — weights FP4 E2M1, activations FP4, scales FP8 E4M3)
Formatcompressed-tensors (native vLLM support)
Toolvllm-project/llm-compressor v0.10.0.2
Calibration128 samples, ultrachat-200k (trainsft split), maxseq_length 2048
Size40 GB (single safetensors shard set)
RequiresNVIDIA Blackwell GPU (SM 120), vLLM >= 0.19

Recipe

yaml
QuantizationModifier:
  targets: [Linear]
  ignore: [lm_head]
  scheme: NVFP4

Usage

vLLM

bash
vllm serve PiehSoft/DeepSeek-R1-Distill-Llama-70B-NVFP4 \
  --host 0.0.0.0 \
  --port 8081 \
  --max-model-len 8192

Python

python
from vllm import LLM, SamplingParams

llm = LLM(model="PiehSoft/DeepSeek-R1-Distill-Llama-70B-NVFP4")
output = llm.generate("What is the meaning of life?", SamplingParams(max_tokens=256))
print(output[0].outputs[0].text)

Benchmarks

Tested on RTX PRO 6000 Blackwell 96GB:

BackendGeneration tok/sPrompt tok/s
vLLM 0.19.025.0176.3
llama.cpp (GGUF variant)33.6196.5

GGUF Version

A GGUF version of this model is available at PiehSoft/DeepSeek-R1-Distill-Llama-70B-NVFP4-GGUF for use with llama.cpp.

Credits

Quantized by PiehSoft (William Pieh) on NVIDIA RTX PRO 6000 Blackwell 96GB.