MoNE-Pruning/DeepSeek-V2-Lite-MoNE-48-zyda2-1000

DeepSeek-V2-Lite-MoNE-48-zyda2-1000

This repository contains a structured pruned variant of DeepSeek-V2-Lite using the MoNE (Mixture-of-Novice Experts) framework proposed in our paper.

*## Model Overview

• —Base Model: DeepSeek-V2-Lite
• —Method: MoNE structured expert pruning
• —Remaining Experts: 48
• —Calibration Set: zyda2-1000
• —Architecture: Mixture-of-Experts (MoE)
• —Framework: Transformers-compatible

This checkpoint replaces redundant experts with lightweight novice experts via structured pruning, aiming to reduce compute while preserving performance.

Paper

Title: MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE Authors: Geng Zhang, Yuxuan Han, Yuxuan Lou, Yiqi Zhang, Wangbo Zhao, Yang You arXiv: arXiv:2507.00390