Modelpublic

jhyuckkim/DeepSeek-V2-Lite-Dense-3B-CP-K6

sourceHugging Faceotherupdated 27d agoView on Hugging Face

0likes554downloads

Model Card

DeepSeek-V2-Lite-Dense-3B-CP-K6

Research artifact, not a general-purpose model. These weights are released so that the experiments in the paper Pruning and Distilling Mixture-of-Experts into Dense Language Models can be reproduced. Each student is distilled on a fixed, small token budget (~4B tokens for Qwen3, 0.3B tokens for DeepSeek-V2-Lite and GPT-OSS) purely so that expert scoring and grouping methods can be compared under an equal budget. Absolute quality is therefore far below the teacher and below pretrained models of the same size, and no instruction tuning or alignment was applied. Please do not use this as an off-the-shelf assistant.

What this is

A dense student obtained by pruning and distilling a Mixture-of-Experts teacher, from the paper *Pruning and Distilling Mixture-of-Experts into Dense Language Models*. Code: https://github.com/krafton-ai/moe-to-dense

Results

Full comparison group for this architecture, DeepSeek-V2-Lite to dense, 0.3B-token distillation (paper Table 17). This model's row is in bold, and rows whose weights are also released link to them.

Configuration	Wino	Hella	ARC-E	ARC-C	MMLU	Avg
SF, K=6	54.9	38.9	52.4	27.1	26.9	40.04
SF, K=12	56.9	41.1	54.8	28.4	24.6	41.16
CP, K=6	53.0	36.9	49.4	25.7	25.3	38.07
CP, K=12	55.6	40.2	53.5	26.6	26.8	40.53
ACP, K=6	56.8	38.6	51.0	27.5	28.1	40.37
ACP, K=12	57.1	40.9	52.9	27.4	26.4	40.93
DO-ACP, K=6	60.3	41.0	53.7	28.2	28.7	42.39
DO-ACP, K=12	59.0	41.5	51.7	26.2	26.9	41.07
Random FFN + teacher attn	50.6	25.6	30.6	20.9	23.6	30.25
Random initialization	50.1	25.4	28.8	24.1	22.9	30.27
Teacher (DeepSeek-V2-Lite)	76.2	80.5	84.4	56.3	58.0	71.09

Downstream accuracy is Winogrande 5-shot, HellaSwag 10-shot, ARC-Easy 25-shot, ARC-Challenge 25-shot and MMLU 5-shot. Avg is the unweighted mean of the five benchmarks.

Configuration

Field	Value
Teacher	`deepseek-ai/DeepSeek-V2-Lite`
Student parameters	2.66B
Distillation data	FineWeb-Edu (`sample-10BT`), 0.3B tokens
Expert scoring	CP
Experts kept (K)	6
Grouping	pure pruning on the routed experts, 8 groups total (2 shared + 6 routed)
Down-projection scaling	shared experts copied unscaled, routed groups scaled by average conditional probability

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "jhyuckkim/DeepSeek-V2-Lite-Dense-3B-CP-K6", dtype="bfloat16", trust_remote_code=True
)
tok = AutoTokenizer.from_pretrained("jhyuckkim/DeepSeek-V2-Lite-Dense-3B-CP-K6", trust_remote_code=True)

trust_remote_code=True is required. The student keeps DeepSeek's own model class so that MLA attention and the shared experts are preserved unchanged, and configuration_deepseek.py / modeling_deepseek.py ship with the repo.

Citation

bibtex

@article{kim2026pruning,
  title={Pruning and Distilling Mixture-of-Experts into Dense Language Models},
  author={Kim, Junhyuck and Yun, Jihun and Kim, Haechan and Kim, Gyeongman and Bae, Joonghyun and Cho, Jaewoong},
  journal={arXiv preprint arXiv:2605.28207},
  year={2026}
}