Modelpublic

thesreedath/slm125m-live-base

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes16downloads

Model Card

thesreedath/slm125m-live-base

A 125M-parameter Llama-architecture small language model pretrained from scratch on a legal/financial corpus (US case law + SEC filings + a fineweb-edu web slice), built with the slm125mLIVE pipeline.

—Params: ~125.8M (12L / 768d / 12h, seq 1024, tied embeddings)
—Tokenizer: 16,384 byte-level BPE trained on the corpus
—Training tokens: ~2.50B (~1 epoch over ~2.19B unique)
—Final val loss: 2.3132250440120696 | val ppl: 10.106967565875875
—Mix: ~40% case-law / ~40% SEC / ~20% web