AverageMetaheuristicsEnjoyer/deepseek-v2-lite-sparse-mobe-b4-bs16-minimal-lr02-h100

sparsedeepseekv2liteb4bs16minimal_lr02

Sparse-MoBE stripped of LISTA priors (T=0, S=0, square encoder, random w-init) on DeepSeek-V2-Lite-Chat, at its own Block-J LR optimum (lr=0.02). Compared against MoBE mobedeepseekv2lite (lr=0.07). Layer 0 is dense (firstkdensereplace=1) so startlayer=1. See docs/sparsemobe_mechanism.md.

• —Method: sparse_mobe
• —Base model: deepseek-ai/DeepSeek-V2-Lite-Chat
• —Compressed: gate_proj, up_proj over layers 1-27 (down_proj kept dense)
• —Settings: sparsemobe, numB=4, activation=tanh

See config.toml and training.json for the full run configuration and per-layer reconstruction stats.

Evaluation

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "AverageMetaheuristicsEnjoyer/deepseek-v2-lite-sparse-mobe-b4-bs16-minimal-lr02-h100",
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto",
)
tok = AutoTokenizer.from_pretrained("AverageMetaheuristicsEnjoyer/deepseek-v2-lite-sparse-mobe-b4-bs16-minimal-lr02-h100", trust_remote_code=True)

The factored experts are reconstructed at runtime via the bundled DeepseekV2SparseMoBEForCausalLM.