Tokenization, Robustness, LLMs
TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
Evaluate models on multiple-choice questions
Compare tokenizers to split text into tokens