--- language: iu language_name: Inuktitut language_family: eskimoaleut tags: - wikilangs - nlp - tokenizer - embeddings - n-gram - markov - wikipedia - feature-extraction - sentence-similarity - tokenization - n-grams - markov-chain - text-mining - fasttext - babelvec - vocabulous - vocabulary - monolingual - family-eskimoaleut license: mit library_name: wikilangs pipeline_tag: text-generation datasets: - omarkamali/wikipedia-monthly dataset_info: name: wikipedia-monthly description: Monthly snapshots of Wikipedia articles across 300+ languages metrics: - name: best_compression_ratio type: compression value: 3.905 - name: best_isotropy type: isotropy value: 0.2183 - name: vocabulary_size type: vocab value: 0 generated: 2026-01-10 --- # Inuktitut - Wikilangs Models ## Comprehensive Research Report & Full Ablation Study This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Inuktitut** Wikipedia data. We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings. ## ๐Ÿ“‹ Repository Contents ### Models & Assets - Tokenizers (8k, 16k, 32k, 64k) - N-gram models (2, 3, 4, 5-gram) - Markov chains (context of 1, 2, 3, 4 and 5) - Subword N-gram and Markov chains - Embeddings in various sizes and dimensions (aligned and unaligned) - Language Vocabulary - Language Statistics ![Performance Dashboard](visualizations/performance_dashboard.png) ### Analysis and Evaluation - [1. Tokenizer Evaluation](#1-tokenizer-evaluation) - [2. N-gram Model Evaluation](#2-n-gram-model-evaluation) - [3. Markov Chain Evaluation](#3-markov-chain-evaluation) - [4. Vocabulary Analysis](#4-vocabulary-analysis) - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation) - [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental) - [7. Summary & Recommendations](#7-summary--recommendations) - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide) - [Visualizations Index](#visualizations-index) --- ## 1. Tokenizer Evaluation ![Tokenizer Compression](visualizations/tokenizer_compression.png) ![Tokenizer Fertility](visualizations/tokenizer_fertility.png) ![Tokenizer OOV](visualizations/tokenizer_oov.png) ![Total Tokens](visualizations/tokenizer_total_tokens.png) ### Results | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens | |------------|-------------|---------------|----------|--------------| | **8k** | 3.015x | 3.02 | 0.1769% | 75,744 | | **16k** | 3.468x | 3.47 | 0.2035% | 65,854 | | **32k** | 3.905x ๐Ÿ† | 3.91 | 0.2292% | 58,476 | ### Tokenization Examples Below are sample sentences tokenized with each vocabulary size: **Sample 1:** `แ•™แƒแ”…แณแ’ƒ แ’ฅแŠแ“•แ’แƒแ‘ฆ แ“„แ“‡แ–“แ“แ“‚ แ–ƒแ•†แ‘•แ…แ”ญแ’ƒแ‘ฏแ‘ฆ แ‘แ‘ญแ“ฏแ’‹แŠแ•แ••แ’ƒ แ“ดแ–…แ‘ญแ‘•แ…แ“šแ…แ–…แ“ฏแ’ชแ”ชแ–… แ’ซแ’ƒ แ“ตแ‘ฏแดแ’กแ’งแ‘ฆ. แ•™แƒแ”…แณแ’ƒ แ‘แ“ดแ…แ’ชแ”ญแ…แ“‚แ–…แนแ–‘แ•—แ–… ...` | Vocab | Tokens | Count | |-------|--------|-------| | 8k | `โ–แ•™แƒแ”…แณแ’ƒ โ–แ’ฅแŠแ“•แ’แƒแ‘ฆ โ–แ“„แ“‡แ–“แ“แ“‚ โ–แ–ƒแ•†แ‘•แ…แ”ญแ’ƒแ‘ฏแ‘ฆ โ–แ‘แ‘ญแ“ฏแ’‹แŠ แ•แ••แ’ƒ โ–แ“ดแ–…แ‘ญแ‘•แ…แ“šแ…แ–…แ“ฏแ’ชแ”ชแ–… โ–แ’ซแ’ƒ โ–แ“ตแ‘ฏ แดแ’ก ... (+16 more)` | 26 | | 16k | `โ–แ•™แƒแ”…แณแ’ƒ โ–แ’ฅแŠแ“•แ’แƒแ‘ฆ โ–แ“„แ“‡แ–“แ“แ“‚ โ–แ–ƒแ•†แ‘•แ…แ”ญแ’ƒแ‘ฏแ‘ฆ โ–แ‘แ‘ญแ“ฏแ’‹แŠ แ•แ••แ’ƒ โ–แ“ดแ–…แ‘ญแ‘•แ…แ“šแ…แ–…แ“ฏแ’ชแ”ชแ–… โ–แ’ซแ’ƒ โ–แ“ตแ‘ฏแดแ’กแ’งแ‘ฆ . ... (+10 more)` | 20 | | 32k | `โ–แ•™แƒแ”…แณแ’ƒ โ–แ’ฅแŠแ“•แ’แƒแ‘ฆ โ–แ“„แ“‡แ–“แ“แ“‚ โ–แ–ƒแ•†แ‘•แ…แ”ญแ’ƒแ‘ฏแ‘ฆ โ–แ‘แ‘ญแ“ฏแ’‹แŠแ•แ••แ’ƒ โ–แ“ดแ–…แ‘ญแ‘•แ…แ“šแ…แ–…แ“ฏแ’ชแ”ชแ–… โ–แ’ซแ’ƒ โ–แ“ตแ‘ฏแดแ’กแ’งแ‘ฆ . โ–แ•™แƒแ”…แณแ’ƒ ... (+7 more)` | 17 | **Sample 2:** `แ…แ“ตแƒแ…โ€”[แ–ƒแ“ชแ“—แ“ˆแ‘Žแ‘แ‘ฆโ€”Ohio]โ€” ) แƒแ‘ŽแŠแ”ชแ‘ฆ แƒแ“—แŠแ“‚. แ…แ“ตแƒแ… แƒแ“„แ–แ‘Ž แŠแ’ฅแŠแ“•แ‘ฒ. แŠแ…แ“šแ‘ฆแ‘Žแ”ฉแ‘ฆ แ“ฏแ•—แ“•แ–…แ‘Žแ–“แ‘ฆ-แ“„แ“‡แ“–แ‘ฆ แ‘ฐแ•‰แ’ปแดแ”… ยซ...` | Vocab | Tokens | Count | |-------|--------|-------| | 8k | `โ–แ…แ“ตแƒแ… โ€”[ แ–ƒแ“ชแ“—แ“ˆแ‘Žแ‘แ‘ฆ โ€” ohio ]โ€” โ–) โ–แƒแ‘ŽแŠแ”ชแ‘ฆ โ–แƒแ“—แŠแ“‚ . ... (+27 more)` | 37 | | 16k | `โ–แ…แ“ตแƒแ… โ€”[ แ–ƒแ“ชแ“—แ“ˆแ‘Žแ‘แ‘ฆ โ€” ohio ]โ€” โ–) โ–แƒแ‘ŽแŠแ”ชแ‘ฆ โ–แƒแ“—แŠแ“‚ . ... (+22 more)` | 32 | | 32k | `โ–แ…แ“ตแƒแ… โ€”[ แ–ƒแ“ชแ“—แ“ˆแ‘Žแ‘แ‘ฆ โ€” ohio ]โ€” โ–) โ–แƒแ‘ŽแŠแ”ชแ‘ฆ โ–แƒแ“—แŠแ“‚ . ... (+22 more)` | 32 | **Sample 3:** `แŠแ…แ‘ฆแ“ฏแ“‡แ–…แ‘แ–… แ“ฑแ“•แŠแ–… แŠแ“‚แ–…แธแ“ˆแ–…แ‘‘แ”ญแ–…แ‘แ–… แ…แ“šแฑแ‘‰แน แ“ดแณแ’ปแ’ฅแ•š แ‘Žแ’ฅ. แ…แ‘‰แ”ญแ’ƒแณแ–… แŠแ“แ“„แ•Œแ“‚แ’ƒ` | Vocab | Tokens | Count | |-------|--------|-------| | 8k | `โ–แŠแ…แ‘ฆแ“ฏแ“‡แ–…แ‘แ–… โ–แ“ฑแ“•แŠแ–… โ–แŠแ“‚แ–…แธแ“ˆแ–…แ‘‘แ”ญแ–…แ‘แ–… โ–แ…แ“šแฑ แ‘‰แน โ–แ“ดแณแ’ปแ’ฅแ•š โ–แ‘Žแ’ฅ . โ–แ…แ‘‰แ”ญแ’ƒแณแ–… โ–แŠแ“แ“„แ•Œแ“‚แ’ƒ` | 10 | | 16k | `โ–แŠแ…แ‘ฆแ“ฏแ“‡แ–…แ‘แ–… โ–แ“ฑแ“•แŠแ–… โ–แŠแ“‚แ–…แธแ“ˆแ–…แ‘‘แ”ญแ–…แ‘แ–… โ–แ…แ“šแฑแ‘‰แน โ–แ“ดแณแ’ปแ’ฅแ•š โ–แ‘Žแ’ฅ . โ–แ…แ‘‰แ”ญแ’ƒแณแ–… โ–แŠแ“แ“„แ•Œแ“‚แ’ƒ` | 9 | | 32k | `โ–แŠแ…แ‘ฆแ“ฏแ“‡แ–…แ‘แ–… โ–แ“ฑแ“•แŠแ–… โ–แŠแ“‚แ–…แธแ“ˆแ–…แ‘‘แ”ญแ–…แ‘แ–… โ–แ…แ“šแฑแ‘‰แน โ–แ“ดแณแ’ปแ’ฅแ•š โ–แ‘Žแ’ฅ . โ–แ…แ‘‰แ”ญแ’ƒแณแ–… โ–แŠแ“แ“„แ•Œแ“‚แ’ƒ` | 9 | ### Key Findings - **Best Compression:** 32k achieves 3.905x compression - **Lowest UNK Rate:** 8k with 0.1769% unknown tokens - **Trade-off:** Larger vocabularies improve compression but increase model size - **Recommendation:** 32k vocabulary provides optimal balance for production use --- ## 2. N-gram Model Evaluation ![N-gram Perplexity](visualizations/ngram_perplexity.png) ![N-gram Unique](visualizations/ngram_unique.png) ![N-gram Coverage](visualizations/ngram_coverage.png) ### Results | N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage | |--------|---------|------------|---------|----------------|------------------|-------------------| | **2-gram** | Word | 93 ๐Ÿ† | 6.54 | 126 | 90.8% | 100.0% | | **2-gram** | Subword | 962 | 9.91 | 3,039 | 37.0% | 87.0% | | **3-gram** | Word | 130 | 7.03 | 174 | 73.9% | 100.0% | | **3-gram** | Subword | 5,020 | 12.29 | 12,029 | 15.7% | 49.7% | | **4-gram** | Word | 694 | 9.44 | 794 | 25.0% | 100.0% | | **4-gram** | Subword | 14,093 | 13.78 | 28,526 | 8.8% | 30.5% | | **5-gram** | Word | 607 | 9.25 | 676 | 24.5% | 100.0% | | **5-gram** | Subword | 19,229 | 14.23 | 32,493 | 7.1% | 24.4% | ### Top 5 N-grams by Size **2-grams (Word):** | Rank | N-gram | Count | |------|--------|-------| | 1 | `san marino` | 73 | | 2 | `of the` | 55 | | 3 | `แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ` | 55 | | 4 | `แ‘ญแ’ปแ’งแ‘ฆ แ…แ–…แ“ฏแ–…` | 47 | | 5 | `แ‘•แ•†แ…แ‘‰ แŠแ‘ญแŠแ“‚` | 44 | **3-grams (Word):** | Rank | N-gram | Count | |------|--------|-------| | 1 | `แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ` | 51 | | 2 | `แ‘ญแ’ปแ’งแ‘ฆ แ…แ–…แ“ฏแ–… www` | 30 | | 3 | `แƒแ“„แ–แ‘Ž แŠแ’ฅแŠแ“•แ‘ฒ แŠแ…แ“šแ‘ฆแ‘Žแ”ฉแ‘ฆ` | 22 | | 4 | `แŠแ…แ“šแ‘ฆแ‘Žแ”ฉแ‘ฆ แ“ฏแ•—แ“•แ–…แ‘Žแ–“แ‘ฆ แ“„แ“‡แ“–แ‘ฆ` | 22 | | 5 | `แŠแ’ฅแŠแ“•แ‘ฒ แŠแ…แ“šแ‘ฆแ‘Žแ”ฉแ‘ฆ แ“ฏแ•—แ“•แ–…แ‘Žแ–“แ‘ฆ` | 22 | **4-grams (Word):** | Rank | N-gram | Count | |------|--------|-------| | 1 | `แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ` | 48 | | 2 | `แƒแ“„แ–แ‘Ž แŠแ’ฅแŠแ“•แ‘ฒ แŠแ…แ“šแ‘ฆแ‘Žแ”ฉแ‘ฆ แ“ฏแ•—แ“•แ–…แ‘Žแ–“แ‘ฆ` | 22 | | 3 | `แŠแ’ฅแŠแ“•แ‘ฒ แŠแ…แ“šแ‘ฆแ‘Žแ”ฉแ‘ฆ แ“ฏแ•—แ“•แ–…แ‘Žแ–“แ‘ฆ แ“„แ“‡แ“–แ‘ฆ` | 22 | | 4 | `แ“„แ“‡แ“–แ‘ฆ แ‘ญแ’ปแ’งแ‘ฆ แ…แ–…แ“ฏแ–… www` | 20 | | 5 | `the grand and general` | 10 | **5-grams (Word):** | Rank | N-gram | Count | |------|--------|-------| | 1 | `แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ` | 45 | | 2 | `แƒแ“„แ–แ‘Ž แŠแ’ฅแŠแ“•แ‘ฒ แŠแ…แ“šแ‘ฆแ‘Žแ”ฉแ‘ฆ แ“ฏแ•—แ“•แ–…แ‘Žแ–“แ‘ฆ แ“„แ“‡แ“–แ‘ฆ` | 22 | | 3 | `the grand and general council` | 10 | | 4 | `แ“„แ“‡ frameless upright 0 3` | 7 | | 5 | `o canada we stand on` | 5 | **2-grams (Subword):** | Rank | N-gram | Count | |------|--------|-------| | 1 | `แ‘ฆ _` | 4,757 | | 2 | `_ แŠ` | 3,099 | | 3 | `แ–… _` | 2,694 | | 4 | `_ แƒ` | 2,386 | | 5 | `, _` | 2,385 | **3-grams (Subword):** | Rank | N-gram | Count | |------|--------|-------| | 1 | `แŠ แ’ป แ’ช` | 851 | | 2 | `_ แŠ แ’ป` | 837 | | 3 | `_ แ“„ แ“‡` | 816 | | 4 | `แ“‚ แ’ƒ _` | 784 | | 5 | `แ‘ฆ _ แŠ` | 710 | **4-grams (Subword):** | Rank | N-gram | Count | |------|--------|-------| | 1 | `_ แŠ แ’ป แ’ช` | 833 | | 2 | `แŠ แ’ป แ’ช _` | 420 | | 3 | `แŠ แ’ป แ’ช แ“—` | 407 | | 4 | `แ–… แ‘ แ–… _` | 405 | | 5 | `แ’ป แ’ช แ“— _` | 385 | **5-grams (Subword):** | Rank | N-gram | Count | |------|--------|-------| | 1 | `_ แŠ แ’ป แ’ช _` | 418 | | 2 | `_ แŠ แ’ป แ’ช แ“—` | 400 | | 3 | `แŠ แ’ป แ’ช แ“— _` | 385 | | 4 | `_ t h e _` | 346 | | 5 | `แ‘ฆ _ แŠ แ’ป แ’ช` | 218 | ### Key Findings - **Best Perplexity:** 2-gram (word) with 93 - **Entropy Trend:** Decreases with larger n-grams (more predictable) - **Coverage:** Top-1000 patterns cover ~24% of corpus - **Recommendation:** 4-gram or 5-gram for best predictive performance --- ## 3. Markov Chain Evaluation ![Markov Entropy](visualizations/markov_entropy.png) ![Markov Contexts](visualizations/markov_contexts.png) ![Markov Branching](visualizations/markov_branching.png) ### Results | Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability | |---------|---------|-------------|------------|------------------|-----------------|----------------| | **1** | Word | 0.3388 | 1.265 | 1.76 | 15,002 | 66.1% | | **1** | Subword | 1.4995 | 2.827 | 13.51 | 541 | 0.0% | | **2** | Word | 0.0479 | 1.034 | 1.07 | 26,047 | 95.2% | | **2** | Subword | 0.9813 | 1.974 | 4.39 | 7,301 | 1.9% | | **3** | Word | 0.0129 | 1.009 | 1.02 | 27,517 | 98.7% | | **3** | Subword | 0.5441 | 1.458 | 2.22 | 31,981 | 45.6% | | **4** | Word | 0.0049 ๐Ÿ† | 1.003 | 1.01 | 27,602 | 99.5% | | **4** | Subword | 0.3121 | 1.242 | 1.55 | 70,999 | 68.8% | ### Generated Text Samples (Word-based) Below are text samples generated from each word-based Markov chain model: **Context Size 1:** 1. `แŠแ’ปแ’ช แฑแ•ˆแ–…แ“ฏแŠแ–… แ‘ญแ’ƒแ‘ฏแ‘ฆ แ…แŠแ‘Žแ’Œแ“ฏแ•†แ’ฅแ’ปแ’งแ‘ฆ แ“ดแƒแ“‡แ’ƒแ‘ญแ…แ”ชแ–… แŠแ‘Žแ–ƒแ•แ’ฅแ‘•แ…แ“‚แ–แ“แ“‚แ’ƒ แŠแ“‚แ”จแ–ƒแ•†แ”ชแ‘ฆ แ…แ“‚แ–…แ‘•แ–ƒแ•แ‘•แ…แ”ชแ‘ฆ แ…แ‘Žแ“‡แ…แ–ƒแ‘Žแ’Œแ‘ฆ แฑแ’ปแ’ฅแ•แ’ฅแ…แ‘•แ“—แ‘‰ แ‘ญแ’ปแ’งแ‘ฆ แ…แ–…แ“ฏแ–… www...` 2. `แŠแ’ปแ’ชแ“— แŠแ…แ“šแ“ƒแ‘ฆ แฑแ“•แ•†แ–ƒแ‘Žแ’Œแ–ƒแ‘ฆแ‘•แ–…แ‘แ‘ฆ แ‹แ–…แ‘ญแ…แ’ชแ‘Žแ‘ฆแ‘Žแ“‚แŠแ•แ“—แ“‚ แŠแ–แ•แ•‹แ’ฅแ’ƒ แ…แ“—แ•†แŠแ“‡แ™ฑแ‘ฆแ‘แ’ƒแ‘ฏแ‘ฆ แ“ฒแ•แ“— แ••แ‘แ•†แ‘ฏ แƒแ“‡แ“—แ’ƒแ‘ฒ แƒแ’กแ“—แ“แ“‚ แ“„แ“‡แ–ƒแ–…แณแ‘ฆ แธแ‘แ‘Ž แƒแ“•แ“šแ…แ–…แ‘•แ•‹ แŠแ…แ“š...` 3. `the roman republic the sammarinese fascist government declared war on their passports citation neede...` **Context Size 2:** 1. `san marino appealed to pope boniface viii against the contribution demands by the legate papal gover...` 2. `แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ’ฅแ‘ญแ”ซแ–•แ“‚แ’ƒ แ‘แŠแ‘ŽแŠแ“‚แ’ƒ แŠแ’ปแ’ช แƒแ“›แ“แ“‚แ’ƒแ‘ฏแ‘ฆ แ“ดแ“‡แ”ญแ…แ•™แ’ƒแ–ขแ‘Žแ’ƒ แ’‘แ‘ฒแ’งแ“•แ’งแ‘ฆ แ“ตแ“ชแ“ดแ’งแ‘ฆ แ“‚แ…แ“แ”…แ’งแ‘ฆ แŠแ’ปแ’ช แ“ฏแ“šแ“แ‘แ’งแ‘ฆ แ‘ฏแ•†แ“แ‘ แ’ชแ‘‰แฑ...` 3. `of the european union it is the fifth smallest country in europe after vatican city and state` **Context Size 3:** 1. `แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ’ƒแ‘ฒแ“แ“‚แ–… แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ` 2. `แ‘ญแ’ปแ’งแ‘ฆ แ…แ–…แ“ฏแ–… www sd gov` 3. `แƒแ“„แ–แ‘Ž แŠแ’ฅแŠแ“•แ‘ฒ แŠแ…แ“šแ‘ฆแ‘Žแ”ฉแ‘ฆ แ“ฏแ•—แ“•แ–…แ‘Žแ–“แ‘ฆ แ“„แ“‡แ“–แ‘ฆ แฒแ• แ–ƒแ“ชแ“—แ“ˆแ‘Žแ‘แ‘ฆ pierre แ“„แ“‡แ“–แ‘ฆ แ‘ญแ’ปแ’งแ‘ฆ แ…แ–…แ“ฏแ–… www ok gov` **Context Size 4:** 1. `แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ แ–„แ–“แ’แ‘ฆ` 2. `แŠแ’ฅแŠแ“•แ‘ฒ แŠแ…แ“šแ‘ฆแ‘Žแ”ฉแ‘ฆ แ“ฏแ•—แ“•แ–…แ‘Žแ–“แ‘ฆ แ“„แ“‡แ“–แ‘ฆ แดแ•แ‘ฆแ“›แ“แ‘ฆ แ–ƒแ“ชแ“—แ“ˆแ‘Žแ‘แ‘ฆ portland แ“„แ“‡แ“–แ‘ฆ แ‘ญแ’ปแ’งแ‘ฆ แ…แ–…แ“ฏแ–… www nv gov` 3. `แƒแ“„แ–แ‘Ž แŠแ’ฅแŠแ“•แ‘ฒ แŠแ…แ“šแ‘ฆแ‘Žแ”ฉแ‘ฆ แ“ฏแ•—แ“•แ–…แ‘Žแ–“แ‘ฆ แ“„แ“‡แ“–แ‘ฆ แ“‚แ… แ†แ•แ“–แ“แ”… แ–ƒแ“ชแ“—แ“ˆแ‘Žแ‘แ‘ฆ new orleans แ“„แ“‡แ“–แ‘ฆ แ‘ญแ’ปแ’งแ‘ฆ แ…แ–…แ“ฏแ–… www idaho gov` ### Generated Text Samples (Subword-based) Below are text samples generated from each subword-based Markov chain model: **Context Size 1:** 1. `_แ••แ’ƒ_แ’แ”ชแ‘Žแ“ชแ“—แ‘•แ‘ญแ“ฏแŠแ’ปแ’ช_` 2. `แ–…แ‘แ’ƒ_ontunixiteco` 3. `แ‘ฆแ‘•)_แ‘ฒ,_แ–„แ•แ’ฅแ“ฑแŠแ•ˆแ‘ŽแŠแ’ป` **Context Size 2:** 1. `แ‘ฆ_(แฑแ“šแ•ˆ,_แ‘Žแ‘Žแ“ชแ“—แ“•แ–ƒแ–…แณแ‘ฆ` 2. `_แŠแ…แธแƒแ’ก_แŠแ–•แ“‡แ–…_แ‘ญแ“•แŠแ‘‰_` 3. `แ–…_แ‘•แƒแ‘ฒแ“‚แธ,_แ“„แ“‡แ“–แ‘ฆ_แƒแ’กแ“—` **Context Size 3:** 1. `แŠแ’ปแ’ชแ“—_แ•ฟแ“šแ’ƒ._แ“ดแ“‚แ‘ญแ“—แŠแ•แ’ฅ.` 2. `_แŠแ’ปแ’ช_แƒแ“—แŠแ“ƒแ‘แ“‚._แƒแ“šแ–ƒแ–…แ‘` 3. `_แ“„แ“‡แ–ƒแƒแ“แ“‡แ•†แŠแ“šแ…แ–…แณแ–…_แŠแ•‹แ••` **Context Size 4:** 1. `_แŠแ’ปแ’ช_แ‘Žแ“ดแ’ชแ“‚แ’ƒ_แ“„แ“‡แ’ฅแ…แ‘•แ…แ•—แ‘ฆ` 2. `แŠแ’ปแ’ช_แ‘•แ‘ฏแ‘ฆแ‘ŽแŠแ”ชแƒแ“แ“‡แ•แ’ฅแ’ƒ_แฑแ–ƒ` 3. `แŠแ’ปแ’ชแ“—_แ–แ••แŠแ“ฑแ–•แ“‚แ–…")แƒแ™ฑแ…แ“ฏแ–“` ### Key Findings - **Best Predictability:** Context-4 (word) with 99.5% predictability - **Branching Factor:** Decreases with context size (more deterministic) - **Memory Trade-off:** Larger contexts require more storage (70,999 contexts) - **Recommendation:** Context-3 or Context-4 for text generation --- ## 4. Vocabulary Analysis ![Zipf's Law](visualizations/zipf_law.png) ![Top Words](visualizations/top20_words.png) ![Coverage Curve](visualizations/vocab_coverage.png) ### Statistics | Metric | Value | |--------|-------| | Vocabulary Size | 3,802 | | Total Tokens | 18,925 | | Mean Frequency | 4.98 | | Median Frequency | 2 | | Frequency Std Dev | 13.99 | ### Most Common Words | Rank | Word | Frequency | |------|------|-----------| | 1 | แŠแ’ปแ’ช | 424 | | 2 | แŠแ’ปแ’ชแ“— | 392 | | 3 | the | 353 | | 4 | of | 210 | | 5 | แƒแ“„แƒแ‘ฆ | 139 | | 6 | and | 131 | | 7 | แ…แ•แ•™แ“˜แ“แ“ƒแ‘ฆ | 114 | | 8 | in | 106 | | 9 | แ–ƒแ“ชแ“—แ“ˆแ‘Žแ‘แ‘ฆ | 104 | | 10 | to | 98 | ### Least Common Words (from vocabulary) | Rank | Word | Frequency | |------|------|-----------| | 1 | แ‘•แ‘ฏแ”ญแ’แ–ƒแ•แ••แ…แ”ชแ‘ฆ | 2 | | 2 | แ’ฅแ…แ“ฏแ… | 2 | | 3 | แ“ดแ’ƒแ‘ฏแ‘แ–ƒแ•แ“„แ‘ฆ | 2 | | 4 | แ“ดแ••แ•‹แ”ญแ“„แ‘ฆ | 2 | | 5 | แŠแ’ฅแŠแ–…แ‘•แ…แ“ฏแ’ชแ”ชแ‘ฆ | 2 | | 6 | แ‘ญแŠแ•‹แ’ฅ | 2 | | 7 | แ”จแŠแ“‡แ•† | 2 | | 8 | แ“ดแ“‡แ“แ–‘แŠแ’แƒแ‘ฆ | 2 | | 9 | แ“…แ‘‰แธแ“ชแ“•แŠแ”ชแ“„แ‘ฆ | 2 | | 10 | แ“„แ“‡แ’ฅแ…แ‘•แ“•แ•†แ“‚แ•แ’งแ‘ฆ | 2 | ### Zipf's Law Analysis | Metric | Value | |--------|-------| | Zipf Coefficient | 0.6869 | | Rยฒ (Goodness of Fit) | 0.969855 | | Adherence Quality | **excellent** | ### Coverage Analysis | Top N Words | Coverage | |-------------|----------| | Top 100 | 30.3% | | Top 1,000 | 65.3% | | Top 5,000 | 0.0% | | Top 10,000 | 0.0% | ### Key Findings - **Zipf Compliance:** Rยฒ=0.9699 indicates excellent adherence to Zipf's law - **High Frequency Dominance:** Top 100 words cover 30.3% of corpus - **Long Tail:** -6,198 words needed for remaining 100.0% coverage --- ## 5. Word Embeddings Evaluation ![Embedding Isotropy](visualizations/embedding_isotropy.png) ![Similarity Matrix](visualizations/embedding_similarity.png) ![t-SNE Words](visualizations/tsne_words.png) ![t-SNE Sentences](visualizations/tsne_sentences.png) ### 5.1 Cross-Lingual Alignment ![Alignment Quality](visualizations/embedding_alignment_quality.png) ![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png) ### 5.2 Model Comparison | Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 | |-------|-----------|----------|------------------|---------------|----------------| | **mono_32d** | 32 | 0.2183 | 0.4714 | N/A | N/A | | **mono_64d** | 64 | 0.0445 | 0.4570 | N/A | N/A | | **mono_128d** | 128 | 0.0046 | 0.4821 | N/A | N/A | | **aligned_32d** | 32 | 0.2183 ๐Ÿ† | 0.4659 | 0.0189 | 0.1384 | | **aligned_64d** | 64 | 0.0445 | 0.4550 | 0.0314 | 0.1384 | | **aligned_128d** | 128 | 0.0046 | 0.4794 | 0.0503 | 0.1509 | ### Key Findings - **Best Isotropy:** aligned_32d with 0.2183 (more uniform distribution) - **Semantic Density:** Average pairwise similarity of 0.4685. Lower values indicate better semantic separation. - **Alignment Quality:** Aligned models achieve up to 5.0% R@1 in cross-lingual retrieval. - **Recommendation:** 128d aligned for best cross-lingual performance --- ## 6. Morphological Analysis (Experimental) This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data. ### 6.1 Productivity & Complexity | Metric | Value | Interpretation | Recommendation | |--------|-------|----------------|----------------| | Productivity Index | **5.000** | High morphological productivity | Reliable analysis | | Idiomaticity Gap | **3.097** | High formulaic/idiomatic content | - | ### 6.2 Affix Inventory (Productive Units) These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts. #### Productive Prefixes | Prefix | Examples | |--------|----------| | `-แŠ` | แŠแ‘แ–…แ‘•แ…แ“ฏแ’ชแ”ชแ–…, แŠแ”…แ‘ฆแ•Œแ“•แŠ, แŠแ“‚แ’แ–…แ‘Žแ“ชแ“—แ’‹แ‘ฆ | | `-แƒ` | แƒแ“…แ–ƒแ‘Žแ’Œแ‘ฆ, แƒแ–ƒแ‘ฆแ‘แ–…, แƒแ“ฑ | | `-แ…` | แ…แ“ชแ“—แ“‚แ’ƒ, แ…แ“›แ“ดแ’ฅ, แ…แ‘ญแ…แ–ƒแ“•แ–…แ‘Žแ“ชแ“—แ’‹แ‘ฆ | | `-แ…แ–ƒ` | แ…แ–ƒแ“•แ’ซแ’แ“„แ‘ฆ, แ…แ–ƒแ…แ“ฏแ’ƒแ“ดแ“‚แ–แ‘ฆ, แ…แ–ƒแ…แ“ฏแ•แ–“แ…แ‘Žแ–ƒแ•แ’ชแ‘Žแ‘• | | `-แ“„แ“‡` | แ“„แ“‡แ••แ…แ‘‰, แ“„แ“‡แ–แ‘Žแ–“แ“‚แ’ƒ, แ“„แ“‡แ™ณแŠแ–“ | | `-แ‘•แƒ` | แ‘•แƒแ’ซแ‘ฆแ“ดแƒแ“แ“‡แ–…, แ‘•แƒแ’ƒแ“ฑแ’ชแ“‚, แ‘•แƒแ’ƒแ‘ฏแ“‡แ“‚ | | `-แƒแ“„` | แƒแ“„แ’ƒ, แƒแ“„แ’‹แŠแ“›แ–‘แ“ชแ“—แ“‚, แƒแ“„แ–•แ“‚แ’ƒ | | `-co` | coca, corporate, country | #### Productive Suffixes | Suffix | Examples | |--------|----------| | `-แ‘ฆ` | แ–ƒแ“šแ’ชแ“แ–แ‘‘แ“—แ‘Žแ“˜แ“แ“ƒแ‘ฆ, แƒแ“…แ–ƒแ‘Žแ’Œแ‘ฆ, แฑแ“•แ•†แ‘ฆแ‘ŽแŠแ•แ“‚แ–แ“แ“„แ‘ฆ | | `-แ–…` | แƒแ–ƒแ‘ฆแ‘แ–…, แŠแ‘แ–…แ‘•แ…แ“ฏแ’ชแ”ชแ–…, แ“ฏแ…แ•‹แ–… | | `-แ’ƒ` | แ“ฏแ•—แ“ชแ“•แ–…แนแ’ƒ, แŠแ‘•แ…แ“ฏแ•แ’ฅแ’ƒ, แ…แ“ชแ“—แ“‚แ’ƒ | | `-แ“‚แ’ƒ` | แ…แ“ชแ“—แ“‚แ’ƒ, แ’ฅแ“•แŠแ“แ“‚แ’ƒ, แ“‚แŠแ–แ•แ“‚แ’ƒ | | `-แ‘แ–…` | แƒแ–ƒแ‘ฆแ‘แ–…, แ“ฏแ…แ•‹แ…แ”ฎแ–…แ‘แ–…, แƒแ“…แ“•แ–…แ‘แ–… | | `-แ“„แ‘ฆ` | แฑแ“•แ•†แ‘ฆแ‘ŽแŠแ•แ“‚แ–แ“แ“„แ‘ฆ, แ‘ญแ–‘แ“ชแ“•แ–…แนแ–…แ“ฏแ…แ‘Žแ“„แ‘ฆ, แŠแ‘•แ…แ“ฏแ…แ–ƒแ‘Žแ’Œแ“„แ‘ฆ | | `-แ“‚` | แ“ฏแ“šแ‘–แ“‚, แƒแ“šแ…แ™ฑแ–ฆแ–ขแ“‚, แ–ƒแ“‚แ’‹แ”ญแ–“แ“‚ | | `-t` | aallatqiit, pitquhiinit, anngutikhaqanngittagaangat | ### 6.3 Bound Stems (Lexical Roots) Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid. | Stem | Cohesion | Substitutability | Examples | |------|----------|------------------|----------| | `แ•—แ“ชแ“•แ–…` | 1.82x | 6 contexts | แ“ฏแ•—แ“ชแ“•แ–…, แ“ฏแ•—แ“ชแ“•แ–…แนแ’ƒ, แ“ฏแ•—แ“ชแ“•แ–…แนแ–… | | `แ“ฏแ•—แ“ชแ“•` | 1.82x | 5 contexts | แ“ฏแ•—แ“ชแ“•แ–…, แ“ฏแ•—แ“ชแ“•แ•แ’ฅ, แ“ฏแ•—แ“ชแ“•แ–…แนแ’ƒ | | `แ–…แ“ฏแ’ชแ”ช` | 1.50x | 6 contexts | แ“‡แƒแ“ˆแ–…แ“ฏแ’ชแ”ชแ–…, แ‘Žแ‘Žแ•‹แ–…แ“ฏแ’ชแ”ชแ–…, แ‘Žแ‘Žแ•‹แ–…แ“ฏแ’ชแ”ชแ’ฅ | | `แ“ฏแ’ชแ”ชแ–…` | 1.72x | 4 contexts | แƒแ“šแ“ฏแ’ชแ”ชแ–…, แ“ดแ“‡แ“ฏแ’ชแ”ชแ–…, แ“ดแ–…แ‘ญแ“ฏแ’ชแ”ชแ–… | | `แ–‘แ“ชแ“—แ“‚` | 1.89x | 3 contexts | แ’ฅแ‘ญแ“›แ–‘แ“ชแ“—แ“‚, แŠแ–แ“›แ–‘แ“ชแ“—แ“‚, แŠแ–แ“›แ–‘แ“ชแ“—แ“‚แ“— | ### 6.4 Affix Compatibility (Co-occurrence) This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology. | Prefix | Suffix | Frequency | Examples | |--------|--------|-----------|----------| | `-แŠ` | `-แ‘ฆ` | 61 words | แŠแ“‚แ’แ–…แ‘Žแ“ชแ“—แ’‹แ‘ฆ, แŠแ…แ“šแ‘ฆแ‘ŽแŠแ•ˆแ“แ“ƒแ–…แ‘แ‘ฆ | | `-แƒ` | `-แ–…` | 47 words | แƒแ–ƒแ‘ฆแ‘แ–…, แƒแ“…แ“•แ–…แ‘แ–… | | `-แƒ` | `-แ‘ฆ` | 46 words | แƒแ“…แ–ƒแ‘Žแ’Œแ‘ฆ, แƒแ“ฏแ’แƒแ‘ฆ | | `-แ…` | `-แ‘ฆ` | 41 words | แ…แ‘ญแ…แ–ƒแ“•แ–…แ‘Žแ“ชแ“—แ’‹แ‘ฆ, แ…แ–ƒแ“•แ’ซแ’แ“„แ‘ฆ | | `-แŠ` | `-แ–…` | 37 words | แŠแ‘แ–…แ‘•แ…แ“ฏแ’ชแ”ชแ–…, แŠแ–แ“›แ–‘แ”ชแ–… | | `-แƒ` | `-แ’ƒ` | 33 words | แƒแ“„แ’ƒ, แƒแ“•แ“แ“‚แŠแ•ˆแ‘Žแ’ฅแ’ƒ | | `-แŠ` | `-แ’ƒ` | 24 words | แŠแ‘•แ…แ“ฏแ•แ’ฅแ’ƒ, แŠแ‘ฏแ“•แ••แ’ƒ | | `-แƒ` | `-แ“‚แ’ƒ` | 19 words | แƒแ“„แ–•แ“‚แ’ƒ, แƒแ•แ•‹แ••แ–แ“แ“‚แ’ƒ | | `-แ…` | `-แ–…` | 19 words | แ…แฑแ•แ–“แ–…, แ…แ–ƒแ…แ“ฏแ–… | | `-แŠ` | `-แ“‚` | 17 words | แŠแ‘แ–…แ‘•แ…แ“ชแ“—แ“‚, แŠแ–แ”ชแ’ปแ’ชแ•†แŠแ“˜แ“ชแ“—แ“‚ | ### 6.5 Recursive Morpheme Segmentation Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`). | Word | Suggested Split | Confidence | Stem | |------|-----------------|------------|------| | แ‹แ–…แ‘ญแ’ƒแ“ฏแ’ชแ“‚แ–“แ“„แ‘ฆ | **`แ‹แ–…แ‘ญแ’ƒแ“ฏแ’ชแ“‚แ–“-แ“„แ‘ฆ`** | 4.5 | `แ‹แ–…แ‘ญแ’ƒแ“ฏแ’ชแ“‚แ–“` | | presented | **`present-ed`** | 4.5 | `present` | | uniformed | **`uniform-ed`** | 4.5 | `uniform` | | แ“„แ“‡แ“•แธแ…แ”ญแ–“แ“„แ‘ฆ | **`แ“„แ“‡แ“•แธแ…แ”ญแ–“-แ“„แ‘ฆ`** | 4.5 | `แ“„แ“‡แ“•แธแ…แ”ญแ–“` | | แ‘–แ’ƒแ“ฐแ”ญแƒแ”ญแ•ˆแ‘Žแ‘ฆ | **`แ‘–แ’ƒแ“ฐแ”ญแƒแ”ญแ•ˆแ‘Ž-แ‘ฆ`** | 4.5 | `แ‘–แ’ƒแ“ฐแ”ญแƒแ”ญแ•ˆแ‘Ž` | | แ‘–แ’ƒแ“ฐแ”ญแƒแ”ญแ•ˆแ‘Žแ“„แ‘ฆ | **`แ‘–แ’ƒแ“ฐแ”ญแƒแ”ญแ•ˆแ‘Ž-แ“„แ‘ฆ`** | 4.5 | `แ‘–แ’ƒแ“ฐแ”ญแƒแ”ญแ•ˆแ‘Ž` | | แ‘–แ’ƒแ“ฐแ”ญแƒแ”ญแ•ˆแ‘Žแ“‚แ’ƒ | **`แ‘–แ’ƒแ“ฐแ”ญแƒแ”ญแ•ˆแ‘Ž-แ“‚แ’ƒ`** | 4.5 | `แ‘–แ’ƒแ“ฐแ”ญแƒแ”ญแ•ˆแ‘Ž` | | แ’ซแ“แ‘Žแ••แ…แ“ชแ‘แ’งแ‘ฆ | **`แ’ซแ“แ‘Žแ••แ…แ“ชแ‘-แ’งแ‘ฆ`** | 4.5 | `แ’ซแ“แ‘Žแ••แ…แ“ชแ‘` | | แŠแ••แ‘ฆแ‘แ–…แ“ฏแ’ชแ”ชแ“‚แ‘ฆ | **`แŠแ••แ‘ฆแ‘แ–…แ“ฏแ’ชแ”ชแ“‚-แ‘ฆ`** | 4.5 | `แŠแ••แ‘ฆแ‘แ–…แ“ฏแ’ชแ”ชแ“‚` | | แƒแ“•แ“แ“‚แŠแ•ˆแ‘Žแ’ฅแ’ƒ | **`แƒแ“•แ“แ“‚แŠแ•ˆแ‘Ž-แ’ฅแ’ƒ`** | 4.5 | `แƒแ“•แ“แ“‚แŠแ•ˆแ‘Ž` | | แƒแ“•แ“แ“‚แŠแ–…แ‘Žแ“‚แ’ƒ | **`แƒแ“•แ“แ“‚แŠแ–…แ‘Ž-แ“‚แ’ƒ`** | 4.5 | `แƒแ“•แ“แ“‚แŠแ–…แ‘Ž` | | แ‹แ–…แ‘ญแ’ƒแ“ฏแ’ชแ“‚แ–“แ“‚แ’ƒ | **`แ‹แ–…แ‘ญแ’ƒแ“ฏแ’ชแ“‚แ–“-แ“‚แ’ƒ`** | 4.5 | `แ‹แ–…แ‘ญแ’ƒแ“ฏแ’ชแ“‚แ–“` | | แƒแ“šแ’‹แ”ญแ…แ“•แ–…แ‘แ–… | **`แƒแ“šแ’‹แ”ญแ…แ“•-แ–…-แ‘แ–…`** | 3.0 | `แƒแ“šแ’‹แ”ญแ…แ“•` | | แŠแ’ฅแŠแ–…แ‘•แ…แ“ฏแ’ชแ”ชแ‘ฆ | **`แŠ-แ’ฅแŠแ–…แ‘•แ…แ“ฏแ’ชแ”ช-แ‘ฆ`** | 3.0 | `แ’ฅแŠแ–…แ‘•แ…แ“ฏแ’ชแ”ช` | | แƒแ“„แ‘แƒแ“แ“‡แ•แ“‚แ’ƒ | **`แƒแ“„-แ‘แƒแ“แ“‡แ•-แ“‚แ’ƒ`** | 3.0 | `แ‘แƒแ“แ“‡แ•` | ### 6.6 Linguistic Interpretation > **Automated Insight:** The language Inuktitut shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding. > **Note on Idiomaticity:** The high Idiomaticity Gap suggests a large number of frequent multi-word expressions or formulaic sequences that are statistically distinct from their component parts. --- ## 7. Summary & Recommendations ![Performance Dashboard](visualizations/performance_dashboard.png) ### Production Recommendations | Component | Recommended | Rationale | |-----------|-------------|-----------| | Tokenizer | **32k BPE** | Best compression (3.91x) | | N-gram | **2-gram** | Lowest perplexity (93) | | Markov | **Context-4** | Highest predictability (99.5%) | | Embeddings | **100d** | Balanced semantic capture and isotropy | --- ## Appendix: Metrics Glossary & Interpretation Guide This section provides definitions, intuitions, and guidance for interpreting the metrics used throughout this report. ### Tokenizer Metrics **Compression Ratio** > *Definition:* The ratio of characters to tokens (chars/token). Measures how efficiently the tokenizer represents text. > > *Intuition:* Higher compression means fewer tokens needed to represent the same text, reducing sequence lengths for downstream models. A 3x compression means ~3 characters per token on average. > > *What to seek:* Higher is generally better for efficiency, but extremely high compression may indicate overly aggressive merging that loses morphological information. **Average Token Length (Fertility)** > *Definition:* Mean number of characters per token produced by the tokenizer. > > *Intuition:* Reflects the granularity of tokenization. Longer tokens capture more context but may struggle with rare words; shorter tokens are more flexible but increase sequence length. > > *What to seek:* Balance between 2-5 characters for most languages. Arabic/morphologically-rich languages may benefit from slightly longer tokens. **Unknown Token Rate (OOV Rate)** > *Definition:* Percentage of tokens that map to the unknown/UNK token, indicating words the tokenizer cannot represent. > > *Intuition:* Lower OOV means better vocabulary coverage. High OOV indicates the tokenizer encounters many unseen character sequences. > > *What to seek:* Below 1% is excellent; below 5% is acceptable. BPE tokenizers typically achieve very low OOV due to subword fallback. ### N-gram Model Metrics **Perplexity** > *Definition:* Measures how "surprised" the model is by test data. Mathematically: 2^(cross-entropy). Lower values indicate better prediction. > > *Intuition:* If perplexity is 100, the model is as uncertain as if choosing uniformly among 100 options at each step. A perplexity of 10 means effectively choosing among 10 equally likely options. > > *What to seek:* Lower is better. Perplexity decreases with larger n-grams (more context). Values vary widely by language and corpus size. **Entropy** > *Definition:* Average information content (in bits) needed to encode the next token given the context. Related to perplexity: perplexity = 2^entropy. > > *Intuition:* High entropy means high uncertainty/randomness; low entropy means predictable patterns. Natural language typically has entropy between 1-4 bits per character. > > *What to seek:* Lower entropy indicates more predictable text patterns. Entropy should decrease as n-gram size increases. **Coverage (Top-K)** > *Definition:* Percentage of corpus occurrences explained by the top K most frequent n-grams. > > *Intuition:* High coverage with few patterns indicates repetitive/formulaic text; low coverage suggests diverse vocabulary usage. > > *What to seek:* Depends on use case. For language modeling, moderate coverage (40-60% with top-1000) is typical for natural text. ### Markov Chain Metrics **Average Entropy** > *Definition:* Mean entropy across all contexts, measuring average uncertainty in next-word prediction. > > *Intuition:* Lower entropy means the model is more confident about what comes next. Context-1 has high entropy (many possible next words); Context-4 has low entropy (few likely continuations). > > *What to seek:* Decreasing entropy with larger context sizes. Very low entropy (<0.1) indicates highly deterministic transitions. **Branching Factor** > *Definition:* Average number of unique next tokens observed for each context. > > *Intuition:* High branching = many possible continuations (flexible but uncertain); low branching = few options (predictable but potentially repetitive). > > *What to seek:* Branching factor should decrease with context size. Values near 1.0 indicate nearly deterministic chains. **Predictability** > *Definition:* Derived metric: (1 - normalized_entropy) ร— 100%. Indicates how deterministic the model's predictions are. > > *Intuition:* 100% predictability means the next word is always certain; 0% means completely random. Real text falls between these extremes. > > *What to seek:* Higher predictability for text generation quality, but too high (>98%) may produce repetitive output. ### Vocabulary & Zipf's Law Metrics **Zipf's Coefficient** > *Definition:* The slope of the log-log plot of word frequency vs. rank. Zipf's law predicts this should be approximately -1. > > *Intuition:* A coefficient near -1 indicates the corpus follows natural language patterns where a few words are very common and most words are rare. > > *What to seek:* Values between -0.8 and -1.2 indicate healthy natural language distribution. Deviations may suggest domain-specific or artificial text. **Rยฒ (Coefficient of Determination)** > *Definition:* Measures how well the linear fit explains the frequency-rank relationship. Ranges from 0 to 1. > > *Intuition:* Rยฒ near 1.0 means the data closely follows Zipf's law; lower values indicate deviation from expected word frequency patterns. > > *What to seek:* Rยฒ > 0.95 is excellent; > 0.99 indicates near-perfect Zipf adherence typical of large natural corpora. **Vocabulary Coverage** > *Definition:* Cumulative percentage of corpus tokens accounted for by the top N words. > > *Intuition:* Shows how concentrated word usage is. If top-100 words cover 50% of text, the corpus relies heavily on common words. > > *What to seek:* Top-100 covering 30-50% is typical. Higher coverage indicates more repetitive text; lower suggests richer vocabulary. ### Word Embedding Metrics **Isotropy** > *Definition:* Measures how uniformly distributed vectors are in the embedding space. Computed as the ratio of minimum to maximum singular values. > > *Intuition:* High isotropy (near 1.0) means vectors spread evenly in all directions; low isotropy means vectors cluster in certain directions, reducing expressiveness. > > *What to seek:* Higher isotropy generally indicates better-quality embeddings. Values > 0.1 are reasonable; > 0.3 is good. Lower-dimensional embeddings tend to have higher isotropy. **Average Norm** > *Definition:* Mean magnitude (L2 norm) of word vectors in the embedding space. > > *Intuition:* Indicates the typical "length" of vectors. Consistent norms suggest stable training; high variance may indicate some words are undertrained. > > *What to seek:* Relatively consistent norms across models. The absolute value matters less than consistency (low std deviation). **Cosine Similarity** > *Definition:* Measures angular similarity between vectors, ranging from -1 (opposite) to 1 (identical direction). > > *Intuition:* Words with similar meanings should have high cosine similarity. This is the standard metric for semantic relatedness in embeddings. > > *What to seek:* Semantically related words should score > 0.5; unrelated words should be near 0. Synonyms often score > 0.7. **t-SNE Visualization** > *Definition:* t-Distributed Stochastic Neighbor Embedding - a dimensionality reduction technique that preserves local structure for visualization. > > *Intuition:* Clusters in t-SNE plots indicate groups of semantically related words. Spread indicates vocabulary diversity; tight clusters suggest semantic coherence. > > *What to seek:* Meaningful clusters (e.g., numbers together, verbs together). Avoid over-interpreting distances - t-SNE preserves local, not global, structure. ### General Interpretation Guidelines 1. **Compare within model families:** Metrics are most meaningful when comparing models of the same type (e.g., 8k vs 64k tokenizer). 2. **Consider trade-offs:** Better performance on one metric often comes at the cost of another (e.g., compression vs. OOV rate). 3. **Context matters:** Optimal values depend on downstream tasks. Text generation may prioritize different metrics than classification. 4. **Corpus influence:** All metrics are influenced by corpus characteristics. Wikipedia text differs from social media or literature. 5. **Language-specific patterns:** Morphologically rich languages (like Arabic) may show different optimal ranges than analytic languages. ### Visualizations Index | Visualization | Description | |---------------|-------------| | Tokenizer Compression | Compression ratios by vocabulary size | | Tokenizer Fertility | Average token length by vocabulary | | Tokenizer OOV | Unknown token rates | | Tokenizer Total Tokens | Total tokens by vocabulary | | N-gram Perplexity | Perplexity by n-gram size | | N-gram Entropy | Entropy by n-gram size | | N-gram Coverage | Top pattern coverage | | N-gram Unique | Unique n-gram counts | | Markov Entropy | Entropy by context size | | Markov Branching | Branching factor by context | | Markov Contexts | Unique context counts | | Zipf's Law | Frequency-rank distribution with fit | | Vocab Frequency | Word frequency distribution | | Top 20 Words | Most frequent words | | Vocab Coverage | Cumulative coverage curve | | Embedding Isotropy | Vector space uniformity | | Embedding Norms | Vector magnitude distribution | | Embedding Similarity | Word similarity heatmap | | Nearest Neighbors | Similar words for key terms | | t-SNE Words | 2D word embedding visualization | | t-SNE Sentences | 2D sentence embedding visualization | | Position Encoding | Encoding method comparison | | Model Sizes | Storage requirements | | Performance Dashboard | Comprehensive performance overview | --- ## About This Project ### Data Source Models trained on [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) - a monthly snapshot of Wikipedia articles across 300+ languages. ### Project A project by **[Wikilangs](https://wikilangs.org)** - Open-source NLP models for every Wikipedia language. ### Maintainer [Omar Kamali](https://omarkamali.com) - [Omneity Labs](https://omneitylabs.com) ### Citation If you use these models in your research, please cite: ```bibtex @misc{wikilangs2025, author = {Kamali, Omar}, title = {Wikilangs: Open NLP Models for Wikipedia Languages}, year = {2025}, doi = {10.5281/zenodo.18073153}, publisher = {Zenodo}, url = {https://huggingface.co/wikilangs} institution = {Omneity Labs} } ``` ### License MIT License - Free for academic and commercial use. ### Links - ๐ŸŒ Website: [wikilangs.org](https://wikilangs.org) - ๐Ÿค— Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs) - ๐Ÿ“Š Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) - ๐Ÿ‘ค Author: [Omar Kamali](https://huggingface.co/omarkamali) - ๐Ÿค Sponsor: [Featherless AI](https://featherless.ai) --- *Generated by Wikilangs Models Pipeline* *Report Date: 2026-01-10 04:55:45*