Bias-variance-tradeoff/slm-125m-base

SLM-125M-base

A 125.8M-parameter Llama-style language model pretrained from random weights on a legal + financial corpus. It is a base completer (next-token prediction only), not an instruction-tuned chatbot — give it the start of a sentence and it continues in the legal/financial register.

Built end-to-end on Modal for the Vizuara "SLM from scratch" workshop, replicating the reference pipeline at Vizuara-AI-Lab/slm-125m-from-scratch.

• —🕹️ Live demo: https://slm-125m-site.vercel.app
• —⚡ Inference API: https://dharsourav03--slm-125m-inference-web.modal.run (/generate)

Intended use

Prompt it with the opening of a sentence and it completes the thought in a legal or financial style. It is a demonstration of the full from-scratch pipeline (data → tokenizer → pretraining), not a production model.

Not intended for: factual question answering, chat, instruction following, or any high-stakes legal/financial decision-making. At 125M parameters it holds very little factual knowledge and will confidently produce plausible-sounding but incorrect content.

Model architecture

Maps 1:1 to transformers.LlamaConfig.

Training data

Streamed from HuggingFace, cleaned, deduplicated, and decontaminated. Realized mix (by real tokens):

Pipeline: stream → deterministic rule-based cleaning (line filters, boilerplate strip, repetition/language/OCR gates) → exact + MinHash near-dedup → 13-gram decontamination against CaseHOLD / LexGLUE eval sets → 16K byte-level BPE tokenizer → pack into 1024-token windows (99/1 train/val split). Final corpus: 2.18B train + 22.0M val tokens.

Training procedure

The reference workshop trains ~5 epochs (→ val perplexity 8.50). This checkpoint is a single-epoch run (val perplexity 10.87); more epochs over the same fixed corpus lower perplexity further.

Evaluation

Held-out validation perplexity is exp(mean token-level cross-entropy) over the 1% val split (21,505 windows, ~22.0M tokens):

Validation perplexity: 10.87 (val loss 2.3856).

How to use

The model uses custom special tokens; prepending <|bos|> matches how it was served and gives the best continuations.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Bias-variance-tradeoff/slm-125m-base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float32).eval()

prompt = "The plaintiff alleges that the defendant"
bos = tok.convert_tokens_to_ids("<|bos|>")
eos = tok.convert_tokens_to_ids("<|eos|>")
ids = torch.tensor([[bos] + tok.encode(prompt, add_special_tokens=False)])

out = model.generate(
    ids, max_new_tokens=90, min_new_tokens=40, do_sample=True,
    temperature=0.8, top_k=50, top_p=0.95, repetition_penalty=1.3,
    eos_token_id=eos, pad_token_id=eos,
)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Limitations and bias

• —Base model, not aligned — no instruction tuning, no RLHF, no safety filtering.
• —Not a knowledge base — a 125M model stores only a few tens of MB of usable knowledge; it fabricates citations, statutes, and figures. Do not rely on any factual claim it makes.
• —Domain-skewed — trained ~72% on US legal + SEC text, so it defaults to that register and reflects the biases of those corpora (US-centric law, corporate finance).
• —English only.

Citation / attribution

Trained from scratch as part of the Vizuara AI Labs "SLM from scratch" workshop, following the pipeline in Vizuara-AI-Lab/slm-125m-from-scratch.