| --- |
| language: en |
| tags: |
| - tokenizer |
| - bpe |
| - shell |
| - code |
| - chatml |
| license: apache-2.0 |
| datasets: |
| - bigcode/the-stack-dedup |
| - m-a-p/CodeFeedback-Filtered-Instruction |
| - HuggingFaceH4/helpful-instructions |
| - HuggingFaceFW/fineweb |
| - Magpie-Align/Magpie-Reasoning-150K |
| --- |
| |
| # DwarfGoToken |
|
|
| A compact **BPE tokenizer** (8,192 tokens) designed for tiny language models that need to understand **shell commands, code snippets, and ChatML-formatted conversations**. Built on top of a custom Go pre‑tokenizer that keeps critical shell tokens (`grep`, `chmod`, `2>&1`, `-rf`, …) atomic, avoiding the fragmentation that kills performance on CPU-bound inference. |
|
|
| ## Why 8,192 tokens? |
|
|
| For a small LM (<20M parameters), a large vocabulary (e.g., 64K) wastes the majority of the model’s parameters on the embedding matrix. With `d_model=256`, the embedding here accounts for only **2.1M parameters (~14%)** — the rest goes into the transformer layers, where it matters most. |
|
|
| ## Corpus |
|
|
| | Source | Domain | Lines | |
| |--------|--------|-------| |
| | `bigcode/the-stack-dedup/shell` | Shell | 1,500,000 | |
| | `bigcode/the-stack-dedup/batchfile` | Batch | 500,000 | |
| | `bigcode/the-stack-dedup/python` | Python | 1,000,000 | |
| | `bigcode/the-stack-dedup/c` | C | 500,000 | |
| | `m-a-p/CodeFeedback-Filtered-Instruction` | Code+Instructions | 200,000 | |
| | `HuggingFaceH4/helpful-instructions` | English instructions | 150,000 | |
| | `HuggingFaceFW/fineweb/sample-10BT` | Web English | 300,000 | |
| | `Magpie-Align/Magpie-Reasoning-150K` | Chain-of-Thought | 200,000 | |
|
|
| **Total:** 4,251,427 lines (3.5 GB) — 47% Shell, 40% Code, 9.5% EN, 3.5% CoT. |
|
|
| ## Special tokens (all atomic) |
|
|
| `<s>`, `</s>`, `<unk>`, `<pad>`, `<|system|>`, `<|user|>`, `<|assistant|>`, `<|end|>`, `<|thinking|>`, `<|/thinking|>`, plus **54 Go‑pre‑tokenizer tokens** (e.g., `grep`, `chmod`, `2>&1`, `&&`, `>>`, `-rf`, `--help`). |
|
|
| ## Quick test |
|
|
| ```python |
| from transformers import AutoTokenizer |
| tok = AutoTokenizer.from_pretrained("ThingAI/DwarfGoToken") |
| |
| # Shell commands stay atomic |
| tok.tokenize("find /var/log -name '*.gz' | xargs rm -rf") |
| # → ['find', '/', 'var', '/', 'log', '-n', 'ame', "'", '*.', 'gz', "'", '|', 'xargs', 'rm', '-rf'] |
| |
| # ChatML template |
| tok.tokenize("<|user|>\nCosa fa grep?\n<|end|>\n<|assistant|>\n...") |
| # → ['<|user|>', 'C', 'os', 'a', 'fa', 'grep', '?', '<|end|>', '<|assistant|>', '...'] |
| ``` |
| ## Usage |
| ```python |
| from transformers import AutoTokenizer |
| tokenizer = AutoTokenizer.from_pretrained("ThingAI/DwarfGoToken") |
| ``` |
| ## Intended use |
| *This tokenizer was built to pair with tiny LMs (~10–20M parameters) specialised in command‑line assistance, shell scripting, or code generation. It’s the companion of the Dwarf model family by ThingsAI.* |
| ## License |
| **Apache 2.0** — *use it, modify it, ship it.* |
