Nahom-M/tokens-to-agents-01-ngrams
2
N-Gram Explorer
Interactive companion to Part 1 — N-grams of the Tokens to Agents series: building the LLM stack from first principles, from n-grams to secure AI agents.
Train unigram, bigram and trigram models on financial news, change the smoothing strength, and watch perplexity respond.
The result this demo is about
Laplace smoothing (α = 1) is the standard fix for zero probabilities. On this corpus it makes a trigram model 4.9× worse than counting single words. Tuning α recovers it, and interpolating all three orders beats every single model by 32%.
Interpolated (0.2 / 0.4 / 0.4) → 90.0
Read the full write-up
- 📖 Article — The Textbook Fix That Made My Language Model 5× Worse
- 💻 Code — github.com/NahomMA/tokens-to-agents
- 🌐 nahomcyberai.tech
Data
Financial PhraseBank (Malo et al., 2014), Sentences_50Agree — 4,846 financial news sentences. Sentiment labels are unused; only the text is modelled.
Nahom M. Birhan — PhD student, agentic AI security.