janakhpon/mon_tokenizer at main

mon_tokenizer

1.1 MB

1 contributor

History: 9 commits

janakhpon

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue

9d74fcf 4 months ago

.gitattributes

285 Bytes

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 4 months ago
.gitignore

114 Bytes

feat: simplified mon tokenizer in hf format, updated tags 4 months ago
.python-version

5 Bytes

feat: simplified mon tokenizer in hf format 4 months ago
README.md

2.65 kB

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 4 months ago
convert_to_hf.py

19.3 kB

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 4 months ago
generation_config.json

291 Bytes

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 4 months ago
mon_tokenizer.meta.json

12.9 kB

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 4 months ago
pyproject.toml

1.1 kB

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 4 months ago
sample_usage.py

6.14 kB

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 4 months ago
special_tokens_map.json

552 Bytes

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 4 months ago
test_tokenizer.py

18.2 kB

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 4 months ago
tokenizer.json

278 kB

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 4 months ago
tokenizer.model

338 kB
xet

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 4 months ago
tokenizer_config.json

1.7 kB

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 4 months ago
upload_to_hub.py

18 kB

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 4 months ago
uv.lock

406 kB

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 4 months ago