wikilangs/tyv
0
1---2language: tyv3language_name: Tuvinian4language_family: turkic_siberian5tags:6 - wikilangs7 - nlp8 - tokenizer9 - embeddings10 - n-gram11 - markov12 - wikipedia13 - feature-extraction14 - sentence-similarity15 - tokenization16 - n-grams17 - markov-chain18 - text-mining19 - fasttext20 - babelvec21 - vocabulous22 - vocabulary23 - monolingual24 - family-turkic_siberian25license: mit26library_name: wikilangs27pipeline_tag: text-generation28datasets:29 - omarkamali/wikipedia-monthly30dataset_info:31 name: wikipedia-monthly32 description: Monthly snapshots of Wikipedia articles across 300+ languages33metrics:34 - name: best_compression_ratio35 type: compression36 value: 4.53737 - name: best_isotropy38 type: isotropy39 value: 0.893540 - name: vocabulary_size41 type: vocab42 value: 043generated: 2026-01-1144---45 46# Tuvinian - Wikilangs Models47## Comprehensive Research Report & Full Ablation Study48 49This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Tuvinian** Wikipedia data.50We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.51 52## 📋 Repository Contents53 54### Models & Assets55 56- Tokenizers (8k, 16k, 32k, 64k)57- N-gram models (2, 3, 4, 5-gram)58- Markov chains (context of 1, 2, 3, 4 and 5)59- Subword N-gram and Markov chains60- Embeddings in various sizes and dimensions (aligned and unaligned)61- Language Vocabulary62- Language Statistics63 6465 66### Analysis and Evaluation67 68- [1. Tokenizer Evaluation](#1-tokenizer-evaluation)69- [2. N-gram Model Evaluation](#2-n-gram-model-evaluation)70- [3. Markov Chain Evaluation](#3-markov-chain-evaluation)71- [4. Vocabulary Analysis](#4-vocabulary-analysis)72- [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)73- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)74- [7. Summary & Recommendations](#7-summary--recommendations)75- [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)76- [Visualizations Index](#visualizations-index)77 78---79## 1. Tokenizer Evaluation80 8182 8384 8586 8788 89### Results90 91| Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |92|------------|-------------|---------------|----------|--------------|93| **8k** | 3.594x | 3.60 | 0.0328% | 531,182 |94| **16k** | 3.989x | 3.99 | 0.0364% | 478,519 |95| **32k** | 4.325x | 4.33 | 0.0394% | 441,354 |96| **64k** | 4.537x 🏆 | 4.54 | 0.0414% | 420,702 |97 98### Tokenization Examples99 100Below are sample sentences tokenized with each vocabulary size:101 102**Sample 1:** `120 — илередип болур: 120 (сан) — 119 биле 121 аразында алыс сан. 120 чыл — григ...`103 104| Vocab | Tokens | Count |105|-------|--------|-------|106| 8k | `▁ 1 2 0 ▁— ▁илередип ▁болур : ▁ 1 ... (+30 more)` | 40 |107| 16k | `▁ 1 2 0 ▁— ▁илередип ▁болур : ▁ 1 ... (+30 more)` | 40 |108| 32k | `▁ 1 2 0 ▁— ▁илередип ▁болур : ▁ 1 ... (+29 more)` | 39 |109| 64k | `▁ 1 2 0 ▁— ▁илередип ▁болур : ▁ 1 ... (+29 more)` | 39 |110 111**Sample 2:** `Волонтёр () – кандыг-ла бир мөөрей, шуулган азы улуг байырлалдарга акша-шалың дэ...`112 113| Vocab | Tokens | Count |114|-------|--------|-------|115| 8k | `▁вол онт ёр ▁() ▁– ▁кандыг - ла ▁бир ▁мөөрей ... (+28 more)` | 38 |116| 16k | `▁вол онт ёр ▁() ▁– ▁кандыг - ла ▁бир ▁мөөрей ... (+25 more)` | 35 |117| 32k | `▁вол онтёр ▁() ▁– ▁кандыг - ла ▁бир ▁мөөрей , ... (+21 more)` | 31 |118| 64k | `▁волонтёр ▁() ▁– ▁кандыг - ла ▁бир ▁мөөрей , ▁шуулган ... (+20 more)` | 30 |119 120**Sample 3:** `Хертек, Артур Ойняр-оол-оглу (хх.хх.ххч. тор.) — Күнзегеш аттыг ном үндүрер төпт...`121 122| Vocab | Tokens | Count |123|-------|--------|-------|124| 8k | `▁хертек , ▁артур ▁ойн яр - оол - оглу ▁( ... (+19 more)` | 29 |125| 16k | `▁хертек , ▁артур ▁ойн яр - оол - оглу ▁( ... (+19 more)` | 29 |126| 32k | `▁хертек , ▁артур ▁ойн яр - оол - оглу ▁( ... (+18 more)` | 28 |127| 64k | `▁хертек , ▁артур ▁ойн яр - оол - оглу ▁( ... (+18 more)` | 28 |128 129 130### Key Findings131 132- **Best Compression:** 64k achieves 4.537x compression133- **Lowest UNK Rate:** 8k with 0.0328% unknown tokens134- **Trade-off:** Larger vocabularies improve compression but increase model size135- **Recommendation:** 32k vocabulary provides optimal balance for production use136 137---138## 2. N-gram Model Evaluation139 140141 142143 144145 146### Results147 148| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |149|--------|---------|------------|---------|----------------|------------------|-------------------|150| **2-gram** | Word | 12,431 | 13.60 | 23,023 | 9.7% | 31.7% |151| **2-gram** | Subword | 472 🏆 | 8.88 | 5,348 | 53.6% | 96.7% |152| **3-gram** | Word | 14,165 | 13.79 | 23,322 | 8.4% | 28.5% |153| **3-gram** | Subword | 4,204 | 12.04 | 40,268 | 18.3% | 58.9% |154| **4-gram** | Word | 28,047 | 14.78 | 43,599 | 6.7% | 21.1% |155| **4-gram** | Subword | 21,807 | 14.41 | 186,047 | 9.6% | 30.8% |156| **5-gram** | Word | 20,854 | 14.35 | 32,166 | 8.0% | 23.8% |157| **5-gram** | Subword | 64,567 | 15.98 | 403,526 | 6.4% | 20.6% |158 159### Top 5 N-grams by Size160 161**2-grams (Word):**162 163| Rank | N-gram | Count |164|------|--------|-------|165| 1 | `бир дугаар` | 1,161 |166| 2 | `тыва республиканың` | 859 |167| 3 | `ынчалза даа` | 839 |168| 4 | `күш ажылдың` | 724 |169| 5 | `ссрэ ниң` | 721 |170 171**3-grams (Word):**172 173| Rank | N-gram | Count |174|------|--------|-------|175| 1 | `социалистиг күш ажылдың` | 353 |176| 2 | `күш ажылдың маадыры` | 325 |177| 3 | `дөс тыва дылдың` | 280 |178| 4 | `чылдан чылга чедир` | 279 |179| 5 | `i наука новосибирск` | 268 |180 181**4-grams (Word):**182 183| Rank | N-gram | Count |184|------|--------|-------|185| 1 | `социалистиг күш ажылдың маадыры` | 316 |186| 2 | `том i наука новосибирск` | 268 |187| 3 | `сөстүү словарь том i` | 240 |188| 4 | `словарь том i наука` | 240 |189| 5 | `ссрэ ниң дээди совединиң` | 190 |190 191**5-grams (Word):**192 193| Rank | N-gram | Count |194|------|--------|-------|195| 1 | `сөстүү словарь том i наука` | 240 |196| 2 | `словарь том i наука новосибирск` | 240 |197| 3 | `ссрэ ниң дээди совединиң президиум` | 158 |198| 4 | `дөс тыва дылдың тайлыбыр сөстүү` | 154 |199| 5 | `тыва дылдың тайлыбыр сөстүү словарь` | 137 |200 201**2-grams (Subword):**202 203| Rank | N-gram | Count |204|------|--------|-------|205| 1 | `а р` | 114,452 |206| 2 | `а _` | 112,817 |207| 3 | `а н` | 101,864 |208| 4 | `. _` | 94,390 |209| 5 | `_ к` | 90,647 |210 211**3-grams (Subword):**212 213| Rank | N-gram | Count |214|------|--------|-------|215| 1 | `ы ң _` | 33,971 |216| 2 | `ы л д` | 29,526 |217| 3 | `_ т у` | 28,537 |218| 4 | `д а _` | 28,076 |219| 5 | `т у р` | 27,698 |220 221**4-grams (Subword):**222 223| Rank | N-gram | Count |224|------|--------|-------|225| 1 | `н ы ң _` | 25,621 |226| 2 | `_ т у р` | 23,645 |227| 3 | `_ ч ы л` | 20,602 |228| 4 | `ы л д а` | 19,319 |229| 5 | `_ б о л` | 18,075 |230 231**5-grams (Subword):**232 233| Rank | N-gram | Count |234|------|--------|-------|235| 1 | `_ ч ы л д` | 16,920 |236| 2 | `ч ы л д а` | 12,964 |237| 3 | `п _ т у р` | 12,742 |238| 4 | `_ т у р г` | 12,031 |239| 5 | `б и л е _` | 11,388 |240 241 242### Key Findings243 244- **Best Perplexity:** 2-gram (subword) with 472245- **Entropy Trend:** Decreases with larger n-grams (more predictable)246- **Coverage:** Top-1000 patterns cover ~21% of corpus247- **Recommendation:** 4-gram or 5-gram for best predictive performance248 249---250## 3. Markov Chain Evaluation251 252253 254255 256257 258### Results259 260| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |261|---------|---------|-------------|------------|------------------|-----------------|----------------|262| **1** | Word | 0.5460 | 1.460 | 3.63 | 203,818 | 45.4% |263| **1** | Subword | 0.0398 | 1.028 | 2.20 | 54,956 | 96.0% |264| **2** | Word | 0.1739 | 1.128 | 1.35 | 738,821 | 82.6% |265| **2** | Subword | 0.1135 | 1.082 | 1.59 | 120,839 | 88.7% |266| **3** | Word | 0.0508 | 1.036 | 1.08 | 995,160 | 94.9% |267| **3** | Subword | 0.3477 | 1.272 | 2.28 | 192,258 | 65.2% |268| **4** | Word | 0.0186 🏆 | 1.013 | 1.03 | 1,068,473 | 98.1% |269| **4** | Subword | 0.4488 | 1.365 | 2.19 | 438,273 | 55.1% |270 271### Generated Text Samples (Word-based)272 273Below are text samples generated from each word-based Markov chain model:274 275**Context Size 1:**276 2771. `биле олурар чаа ыяштар тудуп чиир буура даг советтериниң депутадынга соңгуткан чылда россияның элчин...`2782. `деп башкир педагогика институдунуң улуг хем тыва арат республиканың хөй кичээнгейин өөредилге эрткен...`2793. `чылда ол тываның ном үндүрер ажыл агый рынка развлечений игры в турции и 51 март 8`280 281**Context Size 2:**282 2831. `бир дугаар улуг хуралы ооӊ мурнунда турган календарьны эрги санның деп ылгап тодарадыр моол астроном...`2842. `тыва республиканың өөредилге болгаш эртем яамызының хүндүлел бижии за заслуги перед чувашской респуб...`2853. `ынчалза даа чылдарда экономиканың буурааны биле ол ийи эртемниң үндезин шинчилээр чүүлү кижи бир дуг...`286 287**Context Size 3:**288 2891. `социалистиг күш ажылдың маадыры намдары 3 сентябрь чылда көдээ суур гагида гальского района төрүттүн...`2902. `күш ажылдың маадыры атты тывыскан ленин ордени тыпсыры база серп биле молот медальдар продолжала и д...`2913. `дөс тыва дылдың тайлыбыр сөстүү словарь том i наука новосибирск г г г`292 293**Context Size 4:**294 2951. `социалистиг күш ажылдың маадыры намдары 6 октябрь чылда в кишлаке паткинаб бо үеде дарвазского район...`2962. `том i наука новосибирск в в`2973. `сөстүү словарь том i наука новосибирск й й`298 299 300### Generated Text Samples (Subword-based)301 302Below are text samples generated from each subword-based Markov chain model:303 304**Context Size 1:**305 3061. `_4_биң_штүнаннус`3072. `ажы_绨_андезэрган`3083. `рфелар._доюн_тту`309 310**Context Size 2:**311 3121. `аряның_и_учшен._т`3132. `а_эвеспей_кий_урь`3143. `анолга_өөгүдегенг`315 316**Context Size 3:**317 3181. `ың_ордение_памяти_`3192. `ылдайджанның_демде`3203. `_тур;_калгаш_улуг-`321 322**Context Size 4:**323 3241. `ның_саң-хөөн,_өөрүп`3252. `_тура_шөлүглер_атка`3263. `_чылдың_монгуш,_а_«`327 328 329### Key Findings330 331- **Best Predictability:** Context-4 (word) with 98.1% predictability332- **Branching Factor:** Decreases with context size (more deterministic)333- **Memory Trade-off:** Larger contexts require more storage (438,273 contexts)334- **Recommendation:** Context-3 or Context-4 for text generation335 336---337## 4. Vocabulary Analysis338 339340 341342 343344 345### Statistics346 347| Metric | Value |348|--------|-------|349| Vocabulary Size | 62,436 |350| Total Tokens | 1,039,813 |351| Mean Frequency | 16.65 |352| Median Frequency | 3 |353| Frequency Std Dev | 134.66 |354 355### Most Common Words356 357| Rank | Word | Frequency |358|------|------|-----------|359| 1 | биле | 11,983 |360| 2 | деп | 8,474 |361| 3 | чылда | 8,314 |362| 4 | турган | 8,262 |363| 5 | в | 7,660 |364| 6 | болгаш | 7,220 |365| 7 | ол | 7,087 |366| 8 | база | 7,027 |367| 9 | турар | 6,804 |368| 10 | и | 5,739 |369 370### Least Common Words (from vocabulary)371 372| Rank | Word | Frequency |373|------|------|-----------|374| 1 | 𥼊 | 2 |375| 2 | 𥼋 | 2 |376| 3 | 𥼌 | 2 |377| 4 | 𥼍 | 2 |378| 5 | 𥼎 | 2 |379| 6 | 𥼏 | 2 |380| 7 | 361 | 2 |381| 8 | 359 | 2 |382| 9 | moons | 2 |383| 10 | пегас | 2 |384 385### Zipf's Law Analysis386 387| Metric | Value |388|--------|-------|389| Zipf Coefficient | 0.9998 |390| R² (Goodness of Fit) | 0.992471 |391| Adherence Quality | **excellent** |392 393### Coverage Analysis394 395| Top N Words | Coverage |396|-------------|----------|397| Top 100 | 23.1% |398| Top 1,000 | 51.4% |399| Top 5,000 | 72.7% |400| Top 10,000 | 81.0% |401 402### Key Findings403 404- **Zipf Compliance:** R²=0.9925 indicates excellent adherence to Zipf's law405- **High Frequency Dominance:** Top 100 words cover 23.1% of corpus406- **Long Tail:** 52,436 words needed for remaining 19.0% coverage407 408---409## 5. Word Embeddings Evaluation410 411412 413414 415416 417418 419 420### 5.1 Cross-Lingual Alignment421 422423 424425 426 427### 5.2 Model Comparison428 429| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |430|-------|-----------|----------|------------------|---------------|----------------|431| **mono_32d** | 32 | 0.8935 | 0.3132 | N/A | N/A |432| **mono_64d** | 64 | 0.8586 | 0.2437 | N/A | N/A |433| **mono_128d** | 128 | 0.5600 | 0.2029 | N/A | N/A |434| **aligned_32d** | 32 | 0.8935 🏆 | 0.3180 | 0.0200 | 0.1780 |435| **aligned_64d** | 64 | 0.8586 | 0.2406 | 0.0320 | 0.1860 |436| **aligned_128d** | 128 | 0.5600 | 0.2028 | 0.0720 | 0.2540 |437 438### Key Findings439 440- **Best Isotropy:** aligned_32d with 0.8935 (more uniform distribution)441- **Semantic Density:** Average pairwise similarity of 0.2535. Lower values indicate better semantic separation.442- **Alignment Quality:** Aligned models achieve up to 7.2% R@1 in cross-lingual retrieval.443- **Recommendation:** 128d aligned for best cross-lingual performance444 445---446## 6. Morphological Analysis (Experimental)447 448This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.449 450### 6.1 Productivity & Complexity451 452| Metric | Value | Interpretation | Recommendation |453|--------|-------|----------------|----------------|454| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |455| Idiomaticity Gap | **0.159** | Low formulaic content | - |456 457### 6.2 Affix Inventory (Productive Units)458 459These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.460 461#### Productive Prefixes462| Prefix | Examples |463|--------|----------|464| `-с` | сэ, суперлиги, сөөртүр |465| `-к` | колесников, каразымаар, көрейлер |466| `-а` | арыглаашкын, адагылаар, ааска |467| `-д` | диалектилериниң, действительно, давала |468| `-б` | будут, барбас, буддистериниң |469| `-т` | тыважыдып, турачыларын, тиде |470| `-м` | методологиязын, макрон, моего |471| `-ка` | каразымаар, карлык, каттышканының |472 473#### Productive Suffixes474| Suffix | Examples |475|--------|----------|476| `-а` | шипилина, давала, гарсонкуба |477| `-н` | арыглаашкын, яшин, турачыларын |478| `-ң` | оюннарның, диалектилериниң, фадеевтиң |479| `-р` | ынаныр, каразымаар, көрейлер |480| `-ың` | оюннарның, башкызының, чырыктың |481| `-е` | эктинде, привычное, отставке |482| `-ы` | шыгжамыры, будды, опереттазы |483| `-и` | ниитилелдери, суперлиги, офицерлерни |484 485### 6.3 Bound Stems (Lexical Roots)486 487Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.488 489| Stem | Cohesion | Substitutability | Examples |490|------|----------|------------------|----------|491| `ының` | 1.96x | 48 contexts | зының, атының, цзының |492| `угаа` | 2.04x | 40 contexts | чугаа, угаап, угаан |493| `алда` | 1.66x | 93 contexts | валда, алдан, талда |494| `аның` | 1.81x | 57 contexts | чаның, ханың, ааның |495| `иниң` | 1.92x | 43 contexts | зиниң, линиң, ивиниң |496| `азын` | 1.44x | 151 contexts | чазын, назын, сазын |497| `лдар` | 1.51x | 108 contexts | алдар, салдар, холдар |498| `лган` | 1.67x | 66 contexts | алган, клган, салган |499| `ылды` | 1.76x | 49 contexts | кылды, хылды, чылды |500| `ерге` | 1.61x | 67 contexts | берге, терге, серге |501| `урга` | 1.50x | 80 contexts | турга, уурга, ургаш |502| `рган` | 1.47x | 87 contexts | орган, трган, арган |503 504### 6.4 Affix Compatibility (Co-occurrence)505 506This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.507 508| Prefix | Suffix | Frequency | Examples |509|--------|--------|-----------|----------|510| `-к` | `-н` | 97 words | каналын, карьеразын |511| `-к` | `-а` | 96 words | калбаа, кикбокска |512| `-а` | `-а` | 76 words | азыралга, анкарага |513| `-ч` | `-н` | 70 words | чугаазын, чазын |514| `-с` | `-а` | 70 words | сывында, салчака |515| `-к` | `-ң` | 70 words | кохтуң, консерваториязының |516| `-с` | `-ң` | 60 words | сонуургалдарының, сезонунуң |517| `-т` | `-н` | 57 words | тын, турин |518| `-к` | `-е` | 56 words | күштелдиреринге, кезекте |519| `-б` | `-н` | 55 words | бригадазын, бакалаврын |520 521### 6.5 Recursive Morpheme Segmentation522 523Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).524 525| Word | Suggested Split | Confidence | Stem |526|------|-----------------|------------|------|527| археологтары | **`археологт-а-ры`** | 7.5 | `а` |528| үндезинниң | **`үндезин-н-иң`** | 7.5 | `н` |529| картинная | **`картин-н-ая`** | 7.5 | `н` |530| областьаар | **`область-а-ар`** | 7.5 | `а` |531| чедирислам | **`чедирисл-а-м`** | 7.5 | `а` |532| украинада | **`украин-а-да`** | 7.5 | `а` |533| бадырыптар | **`бадырып-т-ар`** | 7.5 | `т` |534| неделяларга | **`неделял-ар-га`** | 7.5 | `ар` |535| делегейнин | **`делегей-н-ин`** | 7.5 | `н` |536| медалдары | **`медал-да-ры`** | 7.5 | `да` |537| областьта | **`область-т-а`** | 7.5 | `т` |538| чурттапкан | **`чурттап-ка-н`** | 7.5 | `ка` |539| загадочная | **`загадоч-н-ая`** | 7.5 | `н` |540| областная | **`област-н-ая`** | 7.5 | `н` |541| сыктывкаре | **`сыктывк-ар-е`** | 7.5 | `ар` |542 543### 6.6 Linguistic Interpretation544 545> **Automated Insight:**546The language Tuvinian shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.547 548---549## 7. Summary & Recommendations550 551552 553### Production Recommendations554 555| Component | Recommended | Rationale |556|-----------|-------------|-----------|557| Tokenizer | **64k BPE** | Best compression (4.54x) |558| N-gram | **2-gram** | Lowest perplexity (472) |559| Markov | **Context-4** | Highest predictability (98.1%) |560| Embeddings | **100d** | Balanced semantic capture and isotropy |561 562 563---564## Appendix: Metrics Glossary & Interpretation Guide565 566This section provides definitions, intuitions, and guidance for interpreting the metrics used throughout this report.567 568### Tokenizer Metrics569 570**Compression Ratio**571> *Definition:* The ratio of characters to tokens (chars/token). Measures how efficiently the tokenizer represents text.572>573> *Intuition:* Higher compression means fewer tokens needed to represent the same text, reducing sequence lengths for downstream models. A 3x compression means ~3 characters per token on average.574>575> *What to seek:* Higher is generally better for efficiency, but extremely high compression may indicate overly aggressive merging that loses morphological information.576 577**Average Token Length (Fertility)**578> *Definition:* Mean number of characters per token produced by the tokenizer.579>580> *Intuition:* Reflects the granularity of tokenization. Longer tokens capture more context but may struggle with rare words; shorter tokens are more flexible but increase sequence length.581>582> *What to seek:* Balance between 2-5 characters for most languages. Arabic/morphologically-rich languages may benefit from slightly longer tokens.583 584**Unknown Token Rate (OOV Rate)**585> *Definition:* Percentage of tokens that map to the unknown/UNK token, indicating words the tokenizer cannot represent.586>587> *Intuition:* Lower OOV means better vocabulary coverage. High OOV indicates the tokenizer encounters many unseen character sequences.588>589> *What to seek:* Below 1% is excellent; below 5% is acceptable. BPE tokenizers typically achieve very low OOV due to subword fallback.590 591### N-gram Model Metrics592 593**Perplexity**594> *Definition:* Measures how "surprised" the model is by test data. Mathematically: 2^(cross-entropy). Lower values indicate better prediction.595>596> *Intuition:* If perplexity is 100, the model is as uncertain as if choosing uniformly among 100 options at each step. A perplexity of 10 means effectively choosing among 10 equally likely options.597>598> *What to seek:* Lower is better. Perplexity decreases with larger n-grams (more context). Values vary widely by language and corpus size.599 600**Entropy**601> *Definition:* Average information content (in bits) needed to encode the next token given the context. Related to perplexity: perplexity = 2^entropy.602>603> *Intuition:* High entropy means high uncertainty/randomness; low entropy means predictable patterns. Natural language typically has entropy between 1-4 bits per character.604>605> *What to seek:* Lower entropy indicates more predictable text patterns. Entropy should decrease as n-gram size increases.606 607**Coverage (Top-K)**608> *Definition:* Percentage of corpus occurrences explained by the top K most frequent n-grams.609>610> *Intuition:* High coverage with few patterns indicates repetitive/formulaic text; low coverage suggests diverse vocabulary usage.611>612> *What to seek:* Depends on use case. For language modeling, moderate coverage (40-60% with top-1000) is typical for natural text.613 614### Markov Chain Metrics615 616**Average Entropy**617> *Definition:* Mean entropy across all contexts, measuring average uncertainty in next-word prediction.618>619> *Intuition:* Lower entropy means the model is more confident about what comes next. Context-1 has high entropy (many possible next words); Context-4 has low entropy (few likely continuations).620>621> *What to seek:* Decreasing entropy with larger context sizes. Very low entropy (<0.1) indicates highly deterministic transitions.622 623**Branching Factor**624> *Definition:* Average number of unique next tokens observed for each context.625>626> *Intuition:* High branching = many possible continuations (flexible but uncertain); low branching = few options (predictable but potentially repetitive).627>628> *What to seek:* Branching factor should decrease with context size. Values near 1.0 indicate nearly deterministic chains.629 630**Predictability**631> *Definition:* Derived metric: (1 - normalized_entropy) × 100%. Indicates how deterministic the model's predictions are.632>633> *Intuition:* 100% predictability means the next word is always certain; 0% means completely random. Real text falls between these extremes.634>635> *What to seek:* Higher predictability for text generation quality, but too high (>98%) may produce repetitive output.636 637### Vocabulary & Zipf's Law Metrics638 639**Zipf's Coefficient**640> *Definition:* The slope of the log-log plot of word frequency vs. rank. Zipf's law predicts this should be approximately -1.641>642> *Intuition:* A coefficient near -1 indicates the corpus follows natural language patterns where a few words are very common and most words are rare.643>644> *What to seek:* Values between -0.8 and -1.2 indicate healthy natural language distribution. Deviations may suggest domain-specific or artificial text.645 646**R² (Coefficient of Determination)**647> *Definition:* Measures how well the linear fit explains the frequency-rank relationship. Ranges from 0 to 1.648>649> *Intuition:* R² near 1.0 means the data closely follows Zipf's law; lower values indicate deviation from expected word frequency patterns.650>651> *What to seek:* R² > 0.95 is excellent; > 0.99 indicates near-perfect Zipf adherence typical of large natural corpora.652 653**Vocabulary Coverage**654> *Definition:* Cumulative percentage of corpus tokens accounted for by the top N words.655>656> *Intuition:* Shows how concentrated word usage is. If top-100 words cover 50% of text, the corpus relies heavily on common words.657>658> *What to seek:* Top-100 covering 30-50% is typical. Higher coverage indicates more repetitive text; lower suggests richer vocabulary.659 660### Word Embedding Metrics661 662**Isotropy**663> *Definition:* Measures how uniformly distributed vectors are in the embedding space. Computed as the ratio of minimum to maximum singular values.664>665> *Intuition:* High isotropy (near 1.0) means vectors spread evenly in all directions; low isotropy means vectors cluster in certain directions, reducing expressiveness.666>667> *What to seek:* Higher isotropy generally indicates better-quality embeddings. Values > 0.1 are reasonable; > 0.3 is good. Lower-dimensional embeddings tend to have higher isotropy.668 669**Average Norm**670> *Definition:* Mean magnitude (L2 norm) of word vectors in the embedding space.671>672> *Intuition:* Indicates the typical "length" of vectors. Consistent norms suggest stable training; high variance may indicate some words are undertrained.673>674> *What to seek:* Relatively consistent norms across models. The absolute value matters less than consistency (low std deviation).675 676**Cosine Similarity**677> *Definition:* Measures angular similarity between vectors, ranging from -1 (opposite) to 1 (identical direction).678>679> *Intuition:* Words with similar meanings should have high cosine similarity. This is the standard metric for semantic relatedness in embeddings.680>681> *What to seek:* Semantically related words should score > 0.5; unrelated words should be near 0. Synonyms often score > 0.7.682 683**t-SNE Visualization**684> *Definition:* t-Distributed Stochastic Neighbor Embedding - a dimensionality reduction technique that preserves local structure for visualization.685>686> *Intuition:* Clusters in t-SNE plots indicate groups of semantically related words. Spread indicates vocabulary diversity; tight clusters suggest semantic coherence.687>688> *What to seek:* Meaningful clusters (e.g., numbers together, verbs together). Avoid over-interpreting distances - t-SNE preserves local, not global, structure.689 690### General Interpretation Guidelines691 6921. **Compare within model families:** Metrics are most meaningful when comparing models of the same type (e.g., 8k vs 64k tokenizer).6932. **Consider trade-offs:** Better performance on one metric often comes at the cost of another (e.g., compression vs. OOV rate).6943. **Context matters:** Optimal values depend on downstream tasks. Text generation may prioritize different metrics than classification.6954. **Corpus influence:** All metrics are influenced by corpus characteristics. Wikipedia text differs from social media or literature.6965. **Language-specific patterns:** Morphologically rich languages (like Arabic) may show different optimal ranges than analytic languages.697 698 699### Visualizations Index700 701| Visualization | Description |702|---------------|-------------|703| Tokenizer Compression | Compression ratios by vocabulary size |704| Tokenizer Fertility | Average token length by vocabulary |705| Tokenizer OOV | Unknown token rates |706| Tokenizer Total Tokens | Total tokens by vocabulary |707| N-gram Perplexity | Perplexity by n-gram size |708| N-gram Entropy | Entropy by n-gram size |709| N-gram Coverage | Top pattern coverage |710| N-gram Unique | Unique n-gram counts |711| Markov Entropy | Entropy by context size |712| Markov Branching | Branching factor by context |713| Markov Contexts | Unique context counts |714| Zipf's Law | Frequency-rank distribution with fit |715| Vocab Frequency | Word frequency distribution |716| Top 20 Words | Most frequent words |717| Vocab Coverage | Cumulative coverage curve |718| Embedding Isotropy | Vector space uniformity |719| Embedding Norms | Vector magnitude distribution |720| Embedding Similarity | Word similarity heatmap |721| Nearest Neighbors | Similar words for key terms |722| t-SNE Words | 2D word embedding visualization |723| t-SNE Sentences | 2D sentence embedding visualization |724| Position Encoding | Encoding method comparison |725| Model Sizes | Storage requirements |726| Performance Dashboard | Comprehensive performance overview |727 728---729## About This Project730 731### Data Source732 733Models trained on [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) - a monthly snapshot of Wikipedia articles across 300+ languages.734 735### Project736 737A project by **[Wikilangs](https://wikilangs.org)** - Open-source NLP models for every Wikipedia language.738 739### Maintainer740 741[Omar Kamali](https://omarkamali.com) - [Omneity Labs](https://omneitylabs.com)742 743### Citation744 745If you use these models in your research, please cite:746 747```bibtex748@misc{wikilangs2025,749 author = {Kamali, Omar},750 title = {Wikilangs: Open NLP Models for Wikipedia Languages},751 year = {2025},752 doi = {10.5281/zenodo.18073153},753 publisher = {Zenodo},754 url = {https://huggingface.co/wikilangs}755 institution = {Omneity Labs}756}757```758 759### License760 761MIT License - Free for academic and commercial use.762 763### Links764 765- 🌐 Website: [wikilangs.org](https://wikilangs.org)766- 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)767- 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)768- 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)769- 🤝 Sponsor: [Featherless AI](https://featherless.ai)770---771*Generated by Wikilangs Models Pipeline*772 773*Report Date: 2026-01-11 02:16:25*774