OpenTransformer/markov-lm-demo

Markov Chain Language Model

Interactive demo of a classical n-gram language model with Modified Kneser-Ney smoothing.

No neural network — this is pure statistical language modelling using n-gram counts and interpolated backoff.

Architecture

• —Model: 5-gram with Modified Kneser-Ney smoothing
• —Training data: 500M tokens from web crawl datasets
• —Storage: GPU hash tables (sorted int64 keys + torch.searchsorted)
• —Inference: Batch-parallel probability computation via binary search

How It Works

1. 1.Count: Track how often every sequence of 1-5 tokens appears in training data
2. 2.Smooth: Apply Modified Kneser-Ney smoothing to handle unseen n-grams
3. 3.Predict: For a given context, compute P(next_token | context) across all orders
4. 4.Sample: Draw from the smoothed distribution with temperature/top-k/top-p

Performance

Links

• —Model weights: OpenTransformer/markov-5gram-500m
• —Organisation: OpenTransformer

Built by OpenTransformers Ltd. Part of AGILLM research.