Research map / Large language models / Language, safety and interpretability
TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
Gül Sena Altıntaş, Malikeh Ehghaghi, Brian Lester, Fengyuan Liu, Wanru Zhao, Marco Ciccone, Colin Raffel
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TokenMonster tokenizer (ungreedy, consistency constraints) instead of larger multilingual tokenizers | 1B LM · TokSuite multilingual perturbations | relative accuracy drop: 0.17 vs 0.22 (XGLM, BLOOM, ByT5) and 0.26-0.27 (Aya, Tekken) | Improved |
| use byte-level ByT5 tokenizer | 1B LM · TokSuite noise perturbations (non-English) | relative accuracy drop: 0.18 | Improved |
| use byte-level ByT5 tokenizer | 1B LM · Flores200 efficiency | subword fertility: 5.79 vs 1.54 (mBERT) | Hurt |
| apply noise perturbations in non-English instead of English | all 14 tokenizers · TokSuite noise | relative accuracy drop: 0.15 -> 0.21 | Hurt |
| apply Unicode styling perturbations | all 14 tokenizers · TokSuite Unicode (EN) | relative accuracy drop: 0.53 | Hurt |
| add NFKC normalization (XGLM) | XGLM · TokSuite Unicode (EN) | relative accuracy drop: 0.11 vs 0.53 average | Improved |
| scale model from 300M to 1B to 7B with fixed tokenizer | Llama-3.2 · TokSuite perturbations (average) | relative accuracy drop: 0.26, 0.26, 0.22 | No clear effect |
| increase tokenizer vocabulary size | tokenizers · TokSuite robustness | relative accuracy drop | No clear effect |
| use multilingual-specialized tokenizer (XGLM) | XGLM · Flores200 | parity: 1.18 average | Improved |
| use mBERT tokenizer | mBERT · Flores200 | subword fertility: 1.54 | Improved |
| train industry-scale models much longer | larger models · TokSuite | robustness | Mixed |
Closest papers by meaning
- Enhancing Large Language Models through Adaptive Tokenizers (NeurIPS 2024 · similarity 0.91)
- The Foundations of Tokenization: Statistical and Computational Concerns (ICLR 2025 · similarity 0.91)
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.90)
- Model-Aware Tokenizer Transfer (NeurIPS 2026 · similarity 0.90)
- Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.89)
- Explaining and Mitigating Crosslingual Tokenizer Inequities (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
