Modelpublic

vtava/SmolLM2-135M-CeNN-Partition-V3

sourceHugging Faceapache-2.0updated 9d agoView on Hugging Face

0likes

Model Card

SmolLM2-135M CeNN Partition V3

Validation-selected `partition_conservative` TinyCeNN Integrated Memory V3 checkpoint over HuggingFaceTB/SmolLM2-135M.

Architecture

—Variant: cenn_partition
—Replaced attention layers: [0, 29]
—Feature dimension: 64
—Block size: 32
—Sink tokens: 4
—Exact base revision: 93efa2f097d58c2a74874c7e644dbc9b0cee75a2
—TinyCeNN source commit: 0efdb79575204e9f02624ef64e0e6286076ab227

The remaining Transformer layers retain standard attention. This is a partial hybrid research checkpoint, not a fully attention-free model.

Held-out evaluation

Context	Perplexity	PPL ratio vs original	Cache ratio	Prefill speedup	Decode speedup
256	16.421	0.9993	0.9857	0.830×	0.893×
512	14.900	1.0021	0.9595	0.849×	0.910×
1024	14.348	1.0034	0.9464	0.829×	0.892×
2048	14.690	1.0093	0.9399	0.830×	0.891×

Ratios below 1.0 are better for perplexity/cache. Speedups above 1.0 are faster. Current PyTorch CeNN kernels are experimental and are not yet optimized like GPU SDPA.

Load

python

import sys
from huggingface_hub import snapshot_download
folder = snapshot_download('vtava/SmolLM2-135M-CeNN-Partition-V3')
sys.path.insert(0, folder)
from load_model import load_model
model, tokenizer = load_model('vtava/SmolLM2-135M-CeNN-Partition-V3')

Reproducibility

—Validation NLL: 2.8096204151709876
—Before-joint validation NLL: 2.815813680489858
—Joint updates: 300
—Trainable TinyCeNN parameters: 124050
—Training precision: bfloat16
—Full benchmark evidence is under benchmark/.

Limitations

This checkpoint uses a limited held-out set and one training seed. It does not establish universal superiority over Transformer attention. Qualitative generations are not benchmark evidence.

Source and licenses

TinyCeNN-LM: https://github.com/vtavakkoli/TinyCeNN-LM Base model: https://huggingface.co/HuggingFaceTB/SmolLM2-135M SmolLM2 is Apache-2.0. TinyCeNN-LM source is MIT; its license copy is included as LICENSE-TinyCeNN-LM.