Modelpublic

hmellor/tiny-random-DeepseekV2ForCausalLM

sourceHugging Faceupdated 2mo agoView on Hugging Face

0likes21kdownloads

Model Card

tiny-random-DeepseekV2ForCausalLM

Randomly initialised DeepseekV2ForCausalLM for testing MLA (multi-head latent attention) and MoE code paths without loading a 29 GiB checkpoint. Used by vLLM's tests/models/transformers/test_backend.py::test_mla.

The attention head dimensions are copied verbatim from deepseek-ai/DeepSeek-V2-Lite-Chat so that kv_lora_rank + qk_rope_head_dim == 576, the latent head size every MLA kernel is built around. Only the width, depth and expert counts are shrunk.

	DeepSeek-V2-Lite-Chat	this model
`kv_lora_rank`	512	512
`qk_nope_head_dim`	128	128
`qk_rope_head_dim`	64	64
`v_head_dim`	128	128
`hidden_size`	2048	256
`num_hidden_layers`	27	4
`num_attention_heads`	16	4
`n_routed_experts`	64	8
`n_shared_experts`	2	2
parameters	15.7B	59.8M

Layer 0 is dense and the rest are MoE (first_k_dense_replace=1), matching the real model. The tokenizer is DeepSeek-V2-Lite-Chat's, unmodified.

The weights are random, so the outputs are meaningless. This is only useful for checking that two implementations agree with each other.