hmellor/tiny-random-DeepseekV2ForCausalLM

1---2library_name: transformers3tags: []4---5 6# tiny-random-DeepseekV2ForCausalLM7 8Randomly initialised `DeepseekV2ForCausalLM` for testing MLA (multi-head latent9attention) and MoE code paths without loading a 29 GiB checkpoint. Used by10vLLM's `tests/models/transformers/test_backend.py::test_mla`.11 12The attention head dimensions are copied verbatim from13[deepseek-ai/DeepSeek-V2-Lite-Chat](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat)14so that `kv_lora_rank + qk_rope_head_dim == 576`, the latent head size every MLA15kernel is built around. Only the width, depth and expert counts are shrunk.16 17|                     | DeepSeek-V2-Lite-Chat | this model |18| ------------------- | --------------------- | ---------- |19| `kv_lora_rank`      | 512                   | 512        |20| `qk_nope_head_dim`  | 128                   | 128        |21| `qk_rope_head_dim`  | 64                    | 64         |22| `v_head_dim`        | 128                   | 128        |23| `hidden_size`       | 2048                  | 256        |24| `num_hidden_layers` | 27                    | 4          |25| `num_attention_heads` | 16                  | 4          |26| `n_routed_experts`  | 64                    | 8          |27| `n_shared_experts`  | 2                     | 2          |28| parameters          | 15.7B                 | 59.8M      |29 30Layer 0 is dense and the rest are MoE (`first_k_dense_replace=1`), matching the31real model. The tokenizer is DeepSeek-V2-Lite-Chat's, unmodified.32 33The weights are random, so the outputs are meaningless. This is only useful for34checking that two implementations agree with each other.35