wikilangs/en

1---2language: en3language_name: English4language_family: germanic_west_anglofrisian5tags:6  - wikilangs7  - nlp8  - tokenizer9  - embeddings10  - n-gram11  - markov12  - wikipedia13  - feature-extraction14  - sentence-similarity15  - tokenization16  - n-grams17  - markov-chain18  - text-mining19  - fasttext20  - babelvec21  - vocabulous22  - vocabulary23  - monolingual24  - family-germanic_west_anglofrisian25license: mit26library_name: wikilangs27pipeline_tag: text-generation28datasets:29  - omarkamali/wikipedia-monthly30dataset_info:31  name: wikipedia-monthly32  description: Monthly snapshots of Wikipedia articles across 300+ languages33metrics:34  - name: best_compression_ratio35    type: compression36    value: 4.69937  - name: best_isotropy38    type: isotropy39    value: 0.769340  - name: vocabulary_size41    type: vocab42    value: 186753743generated: 2026-03-0344---45 46# English — Wikilangs Models47 48Open-source tokenizers, n-gram & Markov language models, vocabulary stats, and word embeddings trained on **English** Wikipedia by [Wikilangs](https://wikilangs.org).49 50🌐 [Language Page](https://wikilangs.org/languages/en/) · 🎮 [Playground](https://wikilangs.org/playground/?lang=en) · 📊 [Full Research Report](RESEARCH_REPORT.md)51 52## Language Samples53 54Example sentences drawn from the English Wikipedia corpus:55 56> Alexander V may refer to: Alexander V of Macedon (died 294 BCE) Antipope Alexander V Alexander V of Imereti57 58> Alfonso IV may refer to: Alfonso IV of León (924–931) Afonso IV of Portugal Alfonso IV of Aragon Alfonso IV of Ribagorza Alfonso IV d'Este Duke of Modena and Regg59 60> Anastasius I or Anastasios I may refer to: Anastasius I Dicorus (–518), Roman emperor Anastasius I of Antioch (died 599), Patriarch of Antioch Pope Anastasius I (died 401), pope61 62> Angula may refer to: Aṅgula, a measure equal to a finger's breadth Eel, a biological order of fish Nahas Angula, former Prime Minister of Namibia Helmut Angula See also Angul (disambiguation)63 64> Two antipopes used the regnal name Victor IV: Antipope Victor IV Antipope Victor IV65 66## Quick Start67 68### Load the Tokenizer69 70```python71import sentencepiece as spm72 73sp = spm.SentencePieceProcessor()74sp.Load("en_tokenizer_32k.model")75 76text = "Albrecht Achilles may refer to: Albrecht III Achilles, Elector of Brandenburg Al"77tokens = sp.EncodeAsPieces(text)78ids    = sp.EncodeAsIds(text)79 80print(tokens)  # subword pieces81print(ids)     # integer ids82 83# Decode back84print(sp.DecodeIds(ids))85```86 87<details>88<summary><b>Tokenization examples (click to expand)</b></summary>89 90**Sample 1:** `Albrecht Achilles may refer to: Albrecht III Achilles, Elector of Brandenburg Al…`91 92| Vocab | Tokens | Count |93|-------|--------|-------|94| 8k | `▁alb recht ▁ach illes ▁may ▁refer ▁to : ▁alb recht … (+27 more)` | 37 |95| 16k | `▁alb recht ▁ach illes ▁may ▁refer ▁to : ▁alb recht … (+26 more)` | 36 |96| 32k | `▁albrecht ▁achilles ▁may ▁refer ▁to : ▁albrecht ▁iii ▁achilles , … (+17 more)` | 27 |97| 64k | `▁albrecht ▁achilles ▁may ▁refer ▁to : ▁albrecht ▁iii ▁achilles , … (+16 more)` | 26 |98 99**Sample 2:** `Alexander V may refer to: Alexander V of Macedon (died 294 BCE) Antipope Alexand…`100 101| Vocab | Tokens | Count |102|-------|--------|-------|103| 8k | `▁alexander ▁v ▁may ▁refer ▁to : ▁alexander ▁v ▁of ▁maced … (+20 more)` | 30 |104| 16k | `▁alexander ▁v ▁may ▁refer ▁to : ▁alexander ▁v ▁of ▁macedon … (+18 more)` | 28 |105| 32k | `▁alexander ▁v ▁may ▁refer ▁to : ▁alexander ▁v ▁of ▁macedon … (+15 more)` | 25 |106| 64k | `▁alexander ▁v ▁may ▁refer ▁to : ▁alexander ▁v ▁of ▁macedon … (+15 more)` | 25 |107 108**Sample 3:** `Two antipopes used the regnal name Victor IV: Antipope Victor IV Antipope Victor…`109 110| Vocab | Tokens | Count |111|-------|--------|-------|112| 8k | `▁two ▁antip op es ▁used ▁the ▁reg nal ▁name ▁victor … (+8 more)` | 18 |113| 16k | `▁two ▁antip opes ▁used ▁the ▁reg nal ▁name ▁victor ▁iv … (+7 more)` | 17 |114| 32k | `▁two ▁antip opes ▁used ▁the ▁regnal ▁name ▁victor ▁iv : … (+6 more)` | 16 |115| 64k | `▁two ▁antipopes ▁used ▁the ▁regnal ▁name ▁victor ▁iv : ▁antipope … (+5 more)` | 15 |116 117</details>118 119### Load Word Embeddings120 121```python122from gensim.models import KeyedVectors123 124# Aligned embeddings (cross-lingual, mapped to English vector space)125wv = KeyedVectors.load("en_embeddings_128d_aligned.kv")126 127similar = wv.most_similar("word", topn=5)128for word, score in similar:129    print(f"  {word}: {score:.3f}")130```131 132### Load N-gram Model133 134```python135import pyarrow.parquet as pq136 137df = pq.read_table("en_3gram_word.parquet").to_pandas()138print(df.head())139```140 141## Models Overview142 143![Performance Dashboard](visualizations/performance_dashboard.png)144 145| Category | Assets |146|----------|--------|147| Tokenizers | BPE at 8k, 16k, 32k, 64k vocab sizes |148| N-gram models | 2 / 3 / 4 / 5-gram (word & subword) |149| Markov chains | Context 1–5 (word & subword) |150| Embeddings | 32d, 64d, 128d — mono & aligned |151| Vocabulary | Full frequency list + Zipf analysis |152| Statistics | Corpus & model statistics JSON |153 154## Metrics Summary155 156| Component | Model | Key Metric | Value |157|-----------|-------|------------|-------|158| Tokenizer | 8k BPE | Compression | 3.84x |159| Tokenizer | 16k BPE | Compression | 4.22x |160| Tokenizer | 32k BPE | Compression | 4.51x |161| Tokenizer | 64k BPE | Compression | 4.70x 🏆 |162| N-gram | 2-gram (subword) | Perplexity | 257 🏆 |163| N-gram | 2-gram (word) | Perplexity | 386,225 |164| N-gram | 3-gram (subword) | Perplexity | 2,180 |165| N-gram | 3-gram (word) | Perplexity | 4,093,782 |166| N-gram | 4-gram (subword) | Perplexity | 12,758 |167| N-gram | 4-gram (word) | Perplexity | 14,465,722 |168| N-gram | 5-gram (subword) | Perplexity | 55,700 |169| N-gram | 5-gram (word) | Perplexity | 12,820,936 |170| Markov | ctx-1 (subword) | Predictability | 0.0% |171| Markov | ctx-1 (word) | Predictability | 6.2% |172| Markov | ctx-2 (subword) | Predictability | 46.4% |173| Markov | ctx-2 (word) | Predictability | 48.3% |174| Markov | ctx-3 (subword) | Predictability | 45.8% |175| Markov | ctx-3 (word) | Predictability | 75.9% |176| Markov | ctx-4 (subword) | Predictability | 36.8% |177| Markov | ctx-4 (word) | Predictability | 89.2% 🏆 |178| Vocabulary | full | Size | 1,867,537 |179| Vocabulary | full | Zipf R² | 0.9862 |180| Embeddings | mono_32d | Isotropy | 0.7693 🏆 |181| Embeddings | mono_64d | Isotropy | 0.7388 |182| Embeddings | mono_128d | Isotropy | 0.6687 |183 184📊 **[Full ablation study, per-model breakdowns, and interpretation guide →](RESEARCH_REPORT.md)**185 186---187 188## About189 190Trained on [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) — monthly snapshots of 300+ Wikipedia languages.191 192A project by **[Wikilangs](https://wikilangs.org)** · Maintainer: [Omar Kamali](https://omarkamali.com) · [Omneity Labs](https://omneitylabs.com)193 194### Citation195 196```bibtex197@misc{wikilangs2025,198  author    = {Kamali, Omar},199  title     = {Wikilangs: Open NLP Models for Wikipedia Languages},200  year      = {2025},201  doi       = {10.5281/zenodo.18073153},202  publisher = {Zenodo},203  url       = {https://huggingface.co/wikilangs},204  institution = {Omneity Labs}205}206```207 208### Links209 210- 🌐 [wikilangs.org](https://wikilangs.org)211- 🌍 [Language page](https://wikilangs.org/languages/en/)212- 🎮 [Playground](https://wikilangs.org/playground/?lang=en)213- 🤗 [HuggingFace models](https://huggingface.co/wikilangs)214- 📊 [wikipedia-monthly dataset](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)215- 👤 [Omar Kamali](https://huggingface.co/omarkamali)216- 🤝 Sponsor: [Featherless AI](https://featherless.ai)217 218**License:** MIT — free for academic and commercial use.219 220---221*Generated by Wikilangs Pipeline · 2026-03-03 22:59:51*222