almaghrabima
/

SARFTokenizer

Text Generation

Model card Files Files and versions

5.09 MB

Ctrl+K

Ctrl+K

1 contributor

History: 32 commits

almaghrabima's picture

Delete morfessor_models

f2c4997 verified about 1 month ago

.gitattributes

1.64 kB
Upload lexicons/lexicon_en.txt with huggingface_hub about 1 month ago
BENCHMARK_REPORT.md

5.15 kB
Upload BENCHMARK_REPORT.md with huggingface_hub about 1 month ago
README.md

6.99 kB
Update README.md about 1 month ago
benchmark_parallel_results.json

5.38 kB
Upload benchmark_parallel_results.json with huggingface_hub about 1 month ago
benchmark_pypi_full.py

11.6 kB
Upload benchmark_pypi_full.py with huggingface_hub about 1 month ago
benchmark_tiktoken_style.py

8.13 kB
Upload benchmark_tiktoken_style.py with huggingface_hub about 1 month ago
morf_map.bin

115 kB
xet

Upload morf_map.bin with huggingface_hub about 1 month ago
requirements.txt

144 Bytes
Upload requirements.txt with huggingface_hub about 1 month ago
special_tokens_map.json

630 Bytes
Upload special_tokens_map.json with huggingface_hub about 1 month ago
test_comprehensive_million.py

32.6 kB
Upload test_comprehensive_million.py with huggingface_hub about 1 month ago
test_comprehensive_results.json

2.54 kB
Upload test_comprehensive_results.json with huggingface_hub about 1 month ago
tokenizer.json

4.9 MB
Upload tokenizer.json with huggingface_hub about 1 month ago
tokenizer_config.json

4.9 kB
Upload tokenizer_config.json with huggingface_hub about 1 month ago