tiny-random/deepseek-v4-fp4-fp8
137
1---2library_name: transformers3base_model:4- deepseek-ai/DeepSeek-V4-Pro5---6 7This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from [deepseek-ai/DeepSeek-V4-Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro).8 9Note:10 - This model follows the quantization scheme of "FP4 + FP8 Mixed": MoE expert parameters use FP4 precision; most other parameters use FP8.11 - Chat template from [this PR](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/discussions/146/files).12 13| File path | Size |14|------|------|15| model.safetensors | 277.3MB |16 17 18### Example usage:19 20- vLLM21 22```bash23# Not fully tested, please report any issues if you find any problems.24model_id=tiny-random/deepseek-v4-fp4-fp825vllm serve $model_id \26 --trust-remote-code \27 --kv-cache-dtype fp8 \28 --block-size 256 \29 --tensor-parallel-size 2 \30 --no-enable-flashinfer-autotune \31 --tokenizer-mode deepseek_v4 \32 --tool-call-parser deepseek_v4 \33 --enable-auto-tool-choice \34 --reasoning-parser deepseek_v4 \35 --speculative-config '{"method":"mtp","num_speculative_tokens":2}'36```37 38- SGLang39 40```bash41# Tested on H20. Please report any issues if you find any problems.42export NVCC_PREPEND_FLAGS=-allow-unsupported-compiler43export NVCC_APPEND_FLAGS=-allow-unsupported-compiler44export SGLANG_OPT_USE_TILELANG_MHC_PRE=045export SGLANG_OPT_USE_TILELANG_MHC_POST=046export SGLANG_OPT_DEEPGEMM_HC_PRENORM=047export SGLANG_OPT_USE_TILELANG_INDEXER=148 49model_id=tiny-random/deepseek-v4-fp4-fp850sglang serve \51 --trust-remote-code \52 --model-path $model_id \53 --tp 2 \54 --moe-runner-backend marlin \55 --fp8-gemm-backend triton \56 --speculative-algorithm EAGLE \57 --speculative-num-steps 3 \58 --speculative-eagle-topk 1 \59 --speculative-num-draft-tokens 4 \60 --mem-fraction-static 0.6 \61 --disable-cuda-graph \62 --disable-custom-all-reduce63```64 65### Codes to create this repo:66 67<details><summary>Click to expand</summary>68 69```python70import json71import hashlib72from pathlib import Path73from typing import Any, Literal, TypedDict74 75import torch76from huggingface_hub import file_exists, hf_hub_download77from safetensors.torch import save_file78from transformers import AutoTokenizer, GenerationConfig79 80source_model_id = "deepseek-ai/DeepSeek-V4-Pro"81save_folder = "/tmp/tiny-random/deepseek-v4-fp4-fp8"82config = {83 "architectures": [84 "DeepseekV4ForCausalLM"85 ],86 "attention_bias": True,87 "attention_dropout": 0.0,88 "bos_token_id": 0,89 "eos_token_id": 1,90 "expert_dtype": "fp4",91 "hc_eps": 1e-06,92 "hc_mult": 4,93 "hc_sinkhorn_iters": 20,94 # SGLang's DSV4 KV-cache layout fixes the non-RoPE portion at 44895 # elements; together with qk_rope_head_dim=64 this must be 512.96 "head_dim": 512,97 "hidden_act": "silu",98 # SGLang's Hopper MXFP4 Marlin path pads hidden_size to 256. Keeping99 # the checkpoint at 128 leaves its per-32 scales at width 4 while the100 # runtime allocates width 8, so make the checkpoint natively compatible.101 "hidden_size": 256,102 "index_head_dim": 128,103 "index_n_heads": 32,104 "index_topk": 1024,105 "initializer_range": 0.02,106 "max_position_embeddings": 1048576,107 "model_type": "deepseek_v4",108 "moe_intermediate_size": 256,109 "n_routed_experts": 128,110 "n_shared_experts": 1,111 "norm_topk_prob": True,112 "num_attention_heads": 4,113 "num_experts_per_tok": 6,114 "num_hidden_layers": 7,115 "num_hash_layers": 3,116 "num_key_value_heads": 1,117 "num_nextn_predict_layers": 1,118 "o_groups": 2,119 "o_lora_rank": 128,120 "q_lora_rank": 128,121 "qk_rope_head_dim": 64,122 "quantization_config": {123 "activation_scheme": "dynamic",124 "fmt": "e4m3",125 "quant_method": "fp8",126 "scale_fmt": "ue8m0",127 "weight_block_size": [128 128,129 128130 ]131 },132 "rms_norm_eps": 1e-06,133 "rope_scaling": {134 "beta_fast": 32,135 "beta_slow": 1,136 "factor": 16,137 "original_max_position_embeddings": 65536,138 "type": "yarn"139 },140 "rope_theta": 10000,141 "routed_scaling_factor": 2.5,142 "scoring_func": "sqrtsoftplus",143 "sliding_window": 128,144 "swiglu_limit": 10.0,145 "tie_word_embeddings": False,146 "topk_method": "noaux_tc",147 "torch_dtype": "bfloat16",148 "transformers_version": "4.57.1",149 "use_cache": True,150 "vocab_size": 129280,151 "compress_rope_theta": 160000,152 "compress_ratios": [128, 128, 4, 128, 4, 128, 4, 0]153}154 155def main():156 torch.manual_seed(42)157 Path(save_folder).mkdir(parents=True, exist_ok=True)158 state_dict = generate(config)159 save_file(state_dict, Path(save_folder) / "model.safetensors")160 with open(Path(save_folder) / "model.safetensors", "rb") as f:161 state_dict = f.read()162 print("Hash: ", hashlib.sha256(state_dict).hexdigest())163 164 with open(Path(save_folder) / "config.json", "w", encoding="utf-8") as f:165 json.dump(config, f, indent=2, ensure_ascii=False)166 167 tokenizer = AutoTokenizer.from_pretrained(168 source_model_id, trust_remote_code=True,169 )170 if file_exists(filename="chat_template.jinja", repo_id=source_model_id, repo_type='model', revision="refs/pr/146"):171 with open(hf_hub_download(172 source_model_id,173 filename="chat_template.jinja",174 repo_type='model',175 revision="refs/pr/146",176 ), 'r', encoding='utf-8') as f:177 tokenizer.chat_template = f.read()178 tokenizer.save_pretrained(save_folder)179 180 generation_config = GenerationConfig.from_pretrained(181 source_model_id, trust_remote_code=True,182 )183 generation_config.save_pretrained(save_folder)184 185BF16 = "torch.bfloat16"186F32 = "torch.float32"187FP8 = "torch.float8_e4m3fn"188SCALE = "torch.float8_e8m0fnu"189I8 = "torch.int8"190I64 = "torch.int64"191 192class TensorSpec(TypedDict):193 shape: list[int]194 dtype: str195 196Config = dict[str, Any]197State = dict[str, TensorSpec]198TensorDict = dict[str, torch.Tensor]199WeightKind = Literal["bf16", "fp8", "fp4"]200 201def initialize(state: State, config: Config, init_bound: float) -> TensorDict:202 tensors: TensorDict = {}203 scale_dtype = torch.float8_e8m0fnu204 fp4_boundaries = torch.tensor([0.25, 0.75, 1.25, 1.75, 2.5, 3.5, 5.0])205 206 def scale_for(207 amax: torch.Tensor, qmax: float, dtype: torch.dtype208 ) -> torch.Tensor:209 scale = amax.float().clamp_min(torch.finfo(torch.float32).tiny) / qmax210 if dtype == scale_dtype:211 scale = torch.pow(2.0, torch.round(torch.log2(scale)))212 return scale.to(dtype)213 214 def fp8_tensor(215 shape: list[int], scale_shape: list[int], dtype: torch.dtype216 ) -> tuple[torch.Tensor, torch.Tensor]:217 block_out, block_in = config["quantization_config"]["weight_block_size"]218 assert shape == [scale_shape[0] * block_out, scale_shape[1] * block_in]219 value = torch.empty(shape, dtype=torch.bfloat16).uniform_(220 -init_bound, init_bound221 )222 blocks = value.view(223 scale_shape[0], block_out, scale_shape[1], block_in224 ).transpose(1, 2)225 scales = scale_for(blocks.abs().amax(dim=(-1, -2)), 448, dtype)226 weight = (227 (blocks.float() / scales.float()[..., None, None])228 .clamp(-448, 448)229 .to(torch.float8_e4m3fn)230 .transpose(1, 2)231 .reshape(shape)232 .contiguous()233 )234 return weight, scales235 236 def fp4_tensors(237 count: int,238 shape: list[int],239 scale_shape: list[int],240 dtype: torch.dtype,241 ) -> tuple[torch.Tensor, torch.Tensor]:242 out_dim, packed_in_dim = shape243 block_out, block_in = config["quantization_config"]["weight_block_size"]244 assert out_dim % block_out == 0245 assert packed_in_dim * 2 % block_in == 0246 assert packed_in_dim == scale_shape[1] * 16247 value = torch.empty(248 count, out_dim, packed_in_dim * 2, dtype=torch.bfloat16249 ).uniform_(-init_bound, init_bound)250 blocks = value.view(count, out_dim, scale_shape[1], 32)251 scales = scale_for(blocks.abs().amax(dim=-1), 6, dtype)252 normalized = (blocks.float() / scales.float()[..., None]).clamp(-6, 6)253 code = torch.bucketize(normalized.abs(), fp4_boundaries)254 code += normalized.signbit() * 8255 code = code.view(count, out_dim, packed_in_dim * 2)256 packed = (code[..., ::2] | (code[..., 1::2] << 4)).to(torch.uint8)257 weight = packed.view(torch.int8).contiguous()258 return weight, scales259 260 dtype_map: dict[str, torch.dtype] = {261 BF16: torch.bfloat16,262 F32: torch.float32,263 FP8: torch.float8_e4m3fn,264 SCALE: scale_dtype,265 I8: torch.int8,266 I64: torch.int64,267 }268 fp4_groups: dict[269 tuple[tuple[int, ...], tuple[int, ...], torch.dtype], list[str]270 ] = {}271 for name, spec in state.items():272 if spec["dtype"] != I8:273 continue274 scale_spec = state[name.replace(".weight", ".scale")]275 key = (276 tuple(spec["shape"]),277 tuple(scale_spec["shape"]),278 dtype_map[scale_spec["dtype"]],279 )280 fp4_groups.setdefault(key, []).append(name)281 282 fp4_cache: dict[str, tuple[torch.Tensor, torch.Tensor]] = {}283 max_batch_elements = 4 * 1024 * 1024284 for (shape_tuple, scale_shape_tuple, dtype), names in fp4_groups.items():285 shape = list(shape_tuple)286 scale_shape = list(scale_shape_tuple)287 logical_elements = shape[0] * shape[1] * 2288 batch_size = max(1, max_batch_elements // logical_elements)289 for start in range(0, len(names), batch_size):290 batch_names = names[start: start + batch_size]291 weights, scales = fp4_tensors(292 len(batch_names), shape, scale_shape, dtype293 )294 for index, name in enumerate(batch_names):295 fp4_cache[name] = (296 weights[index].clone(),297 scales[index].clone(),298 )299 300 for name, spec in state.items():301 if name in tensors:302 continue303 shape, dtype = spec["shape"], dtype_map[spec["dtype"]]304 scale_name = name.replace(".weight", ".scale")305 if spec["dtype"] == FP8:306 scale_spec = state[scale_name]307 scale_type = dtype_map[scale_spec["dtype"]]308 tensors[name], tensors[scale_name] = fp8_tensor(309 shape, scale_spec["shape"], scale_type310 )311 elif spec["dtype"] == I8:312 tensors[name], tensors[scale_name] = fp4_cache.pop(name)313 elif spec["dtype"] == I64:314 tensors[name] = torch.randint(315 config["n_routed_experts"], shape, dtype=dtype316 )317 elif not name.endswith(".scale"):318 tensors[name] = torch.empty(shape, dtype=dtype).uniform_(319 -init_bound, init_bound320 )321 print(f"{name}: {shape} {dtype}", flush=True)322 file_size_by_name = {name: tensor.numel() * tensor.element_size() for name, tensor in tensors.items()}323 total_file_size = sum(file_size_by_name.values())324 k = 20325 topk = sorted(file_size_by_name.items(), key=lambda x: x[1], reverse=True)[:k]326 print(f"File size: {total_file_size / 1024 / 1024:.2f} MB")327 print(f"Top {k} largest tensors:")328 for name, size in topk:329 print(f" {name}: {size / 1024 / 1024:.2f} MB")330 return tensors331 332def generate(333 config: Config, init_bound: float = 0.2334) -> TensorDict:335 assert init_bound > 0336 state: State = {}337 dim = config["hidden_size"]338 inter = config["moe_intermediate_size"]339 heads = config["num_attention_heads"]340 head_dim = config["head_dim"]341 q_rank = config["q_lora_rank"]342 o_rank = config["o_lora_rank"]343 o_groups = config["o_groups"]344 experts = config["n_routed_experts"]345 vocab = config["vocab_size"]346 hc = config["hc_mult"]347 quant = config["quantization_config"]348 block_out, block_in = quant["weight_block_size"]349 weight_kind: WeightKind = (350 "fp8" if quant["quant_method"] == "fp8" else "bf16"351 )352 scale_dtype = SCALE if quant.get("scale_fmt") == "ue8m0" else F32353 354 def add(name: str, shape: list[int], dtype: str) -> None:355 state[name] = {"shape": shape, "dtype": dtype}356 357 def linear(358 name: str, out_dim: int, in_dim: int, kind: WeightKind = weight_kind359 ) -> None:360 if kind == "fp4":361 assert out_dim % block_out == 0 and in_dim % block_in == 0, (362 f"{name} shape [{out_dim}, {in_dim}] is not divisible by "363 f"block size [{block_out}, {block_in}]"364 )365 add(f"{name}.weight", [out_dim, in_dim // 2], I8)366 add(f"{name}.scale", [out_dim, in_dim // 32], scale_dtype)367 elif kind == "fp8":368 assert out_dim % block_out == 0 and in_dim % block_in == 0, (369 f"{name} shape [{out_dim}, {in_dim}] is not divisible by "370 f"block size [{block_out}, {block_in}]"371 )372 add(f"{name}.weight", [out_dim, in_dim], FP8)373 add(374 f"{name}.scale",375 [out_dim // block_out, in_dim // block_in],376 scale_dtype,377 )378 else:379 add(f"{name}.weight", [out_dim, in_dim], BF16)380 381 def compressor(name: str, ratio: int, size: int) -> None:382 out_dim = size * (2 if ratio == 4 else 1)383 add(f"{name}.ape", [ratio, out_dim], F32)384 add(f"{name}.wkv.weight", [out_dim, dim], BF16)385 add(f"{name}.wgate.weight", [out_dim, dim], BF16)386 add(f"{name}.norm.weight", [size], BF16)387 388 def attention(name: str, ratio: int) -> None:389 add(f"{name}.attn_sink", [heads], F32)390 linear(f"{name}.wq_a", q_rank, dim)391 add(f"{name}.q_norm.weight", [q_rank], BF16)392 linear(f"{name}.wq_b", heads * head_dim, q_rank)393 linear(f"{name}.wkv", head_dim, dim)394 add(f"{name}.kv_norm.weight", [head_dim], BF16)395 linear(f"{name}.wo_a", o_groups * o_rank, heads * head_dim // o_groups)396 linear(f"{name}.wo_b", dim, o_groups * o_rank)397 398 if ratio:399 compressor(f"{name}.compressor", ratio, head_dim)400 if ratio == 4:401 index_heads = config["index_n_heads"]402 index_dim = config["index_head_dim"]403 linear(f"{name}.indexer.wq_b", index_heads * index_dim, q_rank)404 add(f"{name}.indexer.weights_proj.weight", [index_heads, dim], BF16)405 compressor(f"{name}.indexer.compressor", ratio, index_dim)406 407 def expert(name: str, kind: WeightKind) -> None:408 linear(f"{name}.w1", inter, dim, kind)409 linear(f"{name}.w2", dim, inter, kind)410 linear(f"{name}.w3", inter, dim, kind)411 412 def moe(name: str, layer_id: int) -> None:413 add(f"{name}.gate.weight", [experts, dim], BF16)414 if layer_id < config["num_hash_layers"]:415 add(416 f"{name}.gate.tid2eid",417 [vocab, config["num_experts_per_tok"]],418 I64,419 )420 else:421 add(f"{name}.gate.bias", [experts], F32)422 423 routed_kind = "fp4" if config.get("expert_dtype") == "fp4" else weight_kind424 for expert_id in range(experts):425 expert(f"{name}.experts.{expert_id}", routed_kind)426 expert(f"{name}.shared_experts", weight_kind)427 428 def block(name: str, layer_id: int) -> None:429 attention(f"{name}.attn", config["compress_ratios"][layer_id])430 moe(f"{name}.ffn", layer_id)431 add(f"{name}.attn_norm.weight", [dim], BF16)432 add(f"{name}.ffn_norm.weight", [dim], BF16)433 for part in ("attn", "ffn"):434 add(f"{name}.hc_{part}_fn", [(2 + hc) * hc, hc * dim], F32)435 add(f"{name}.hc_{part}_base", [(2 + hc) * hc], F32)436 add(f"{name}.hc_{part}_scale", [3], F32)437 438 def hc_head(name: str = "") -> None:439 prefix = f"{name}." if name else ""440 add(f"{prefix}hc_head_fn", [hc, hc * dim], F32)441 add(f"{prefix}hc_head_base", [hc], F32)442 add(f"{prefix}hc_head_scale", [1], F32)443 444 add("embed.weight", [vocab, dim], BF16)445 for layer_id in range(config["num_hidden_layers"]):446 block(f"layers.{layer_id}", layer_id)447 add("norm.weight", [dim], BF16)448 add("head.weight", [vocab, dim], BF16)449 hc_head()450 451 first_mtp_layer = config["num_hidden_layers"]452 for mtp_id in range(config["num_nextn_predict_layers"]):453 name = f"mtp.{mtp_id}"454 block(name, first_mtp_layer + mtp_id)455 linear(f"{name}.e_proj", dim, dim)456 linear(f"{name}.h_proj", dim, dim)457 for norm in ("enorm", "hnorm", "norm"):458 add(f"{name}.{norm}.weight", [dim], BF16)459 hc_head(name)460 461 return initialize(state, config, init_bound)462 463main()464```465 466</details>467 468### Test environment:469 470- safetensors: 0.8.0471- sglang: 0.5.15.post1472- torch: 2.11.0+cu129473- transformers: 5.12.1474 475### Change log:476 477- 2026-07-15: Initial version.478- 2026-07-22: Added SGLang example. Updated model config so that SGLang can run.