CoolFace
Modelpublic

vtava/TinyCeNN-LM-Distilled-v2

sourceHugging Faceupdated 12d agoView on Hugging Face
0likes
Model Card

TinyCeNN-LM Distilled v2

Transformer-free CeNN student distilled from arnir0/Tiny-LLM.

Architecture

  • —Transformer layers remaining: 0
  • —CeNN recurrent steps: 7
  • —CeNN receptive field: 255 tokens
  • —Trainable CeNN parameters: 480,192

Rigorous benchmark

  • —Protocol: rigorous-v2
  • —Deterministic held-out tokens: 65,536
  • —Benchmark SHA256: 319c4527bb6a104bdd3772f30204fc8c32521f355f57d7c7fd387dbb058c3021
  • —Cumulative distillation tokens: 40,005,632
  • —Best student CE: 5.122586
  • —Teacher CE: 4.234912
  • —Best student PPL: 167.769
  • —Teacher PPL: 69.056
  • —Teacher-gap recovery: 84.42%

Load via tinycenn_lm.build_cenn_student().