juiceb0xc0de/smollm2-135m-instruct-SAE
Layer EV Mean L0 Recon Loss Dead % 0 0.9480 48.74 0.2074 0.0 1 0.9599 43.65 0.3298 0.0 2 0.9631 46.81 0.5021 0.0 3 0.9508 46.56 0.7462 0.0 4 0.9463 46.23 0.8936 0.0 5 0.9350 47.57 1.1605 0.0 6 0.9306 48.44 1.3838 0.0 7 0.9318 49.51 1.5446 0.0 8 0.9432 46.52 1.6598 0.0 9 0.9373 47.15 2.0706 0.0 10 0.9348 45.53 2.2983 0.0 11 0.9905 48.58 5.8113 0.0 12 0.9901 48.42 6.1039 0.0 13 0.9891 46.15 6.9692 0.0 14 0.9884 44.76 7.1844 0.0 15 0.9863 47.63 8.6521 0.0… See the full description on the dataset page: https://huggingface.co/datasets/juiceb0xc0de/smollm2-135m-instruct-SAE.
Highlights:
- Dead features: 0.0% across all 30 layers. Not averaged. Every single layer.
- Mid layers 11–18: EV above 0.98, peaking at 0.9905 on layer 11
- L0 stayed tight: 43.45–52.01 against a target of 50
- Mean EV across all layers: 0.9519
SAE training
Target L0 of 50, rolling-HF capture, 500 pool batches, 5000 max steps, 32,768 tokens per microbatch. Custom trainer — built from scratch for this run, not adapted from an existing library. The full training log and per-layer breakdown live in `smollm2-135-instruct-SAE-trainer.md`. The raw Modal terminal output is preserved in `smollm2-135-instruct-SAE-trainer.raw.log`.
Layer 29 needed a separate pass due to an off-by-one in the layer range argument — 0,29 trained layers 0–28, not 0–29. It still converged clean — EV 0.8812, L0 52.01, 0 dead features.