Zaynoid/Kimi-K2-Thinking-Tokenizer

Kimi-K2-Thinking Tokenizer

Tokenizer files for moonshotai/Kimi-K2-Thinking - a trillion-parameter thinking model.

📦 What's Included

• tiktoken.model - Original tiktoken tokenizer (2.8MB)
• tokenizer.json - HuggingFace compatible format
• tokenization_kimi.py - Custom tokenization code
• tokenizer_config.json - Configuration
• special_tokens_map.json - Special tokens
• chat_template.jinja - Chat template

🚀 Quick Start

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "Zaynoid/Kimi-K2-Thinking-Tokenizer",
    trust_remote_code=True
)

text = "Hello, how are you?"
tokens = tokenizer.encode(text)
print(f"Tokens: {tokens}")
print(f"Decoded: {tokenizer.decode(tokens)}")

📊 Specifications

• Vocab Size: 163,840 tokens
• Model Type: Tiktoken (BPE-based)
• Context Length: 256K tokens
• Special Tokens:
  • BOS: <｜begin▁of▁sentence｜> (ID: 163584)
  • EOS: <｜end▁of▁sentence｜> (ID: 163585)

💡 Usage Notes

1. Recommended: Use with trust_remote_code=True for full compatibility
2. The tokenizer.json is provided for tools that require it
3. Original model uses tiktoken format natively

🔧 Use with vLLM

from vllm import LLM

llm = LLM(
    model="moonshotai/Kimi-K2-Thinking",
    tokenizer="Zaynoid/Kimi-K2-Thinking-Tokenizer",
    trust_remote_code=True
)

📝 License

Modified MIT License (same as base model)

🙏 Credits

• Original Model: Moonshot AI
• Architecture: Based on DeepSeek-V3
• Tokenizer Extraction: Zaynoid

🔗 Links

• Base Model
• Model Card
• Moonshot AI