Modelpublic

vtava/TinyCeNN-LM-Sharded-MoE-Top2

sourceHugging Faceupdated 12d agoView on Hugging Face

0likes

Model Card

TinyCeNN-LM Sharded MoE Top-2

Transformer-free CeNN student using a parameter-neutral routed FFN.

—8 shards × 96 inner channels
—Top-2 routing
—trainable parameters: 481,729
—overhead vs plain CeNN: 0.320%
—held-out tokens: 65,536
—best CE: 5.115179
—teacher CE: 4.234912
—teacher-gap recovery: 84.55%