Research map / Large language models / Language, safety and interpretability
MoTo: Mixture of Tokenizers Towards Fair Multilingual Language Modeling
Gül Sena Altıntaş, Colin Raffel
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mixture of Tokenizers (MoTo), a modular tokenization approach for multilingual language models aimed at fairer treatment of morphologically rich and non-Latin-script languages.
Problem
Single, English-dominant tokenizers fixed at training time fragment morphologically rich and non-Latin-script languages, producing longer sequences and lower information density, which hurts multilingual fairness.
Approach
A modular mixture of tokenizers that combines multiple tokenizers rather than relying on one fixed tokenizer chosen at training time; abstract is truncated so specifics are unknown.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use language-specific per-language tokenizers (8K each) instead of one Llama-3 tokenizer | MoTo · English benchmarks, 1B model | accuracy relative to Llama-3: +0.54 ± 0.38 | No clear effect |
| use per-language tokenizers instead of Llama tokenizer | MoTo · multilingual downstream tasks, 20 non-English languages | accuracy | Improved |
| use jointly trained mBPE instead of per-language tokenizers (ablation) | mBPE (2K/42K) · synthetic multilingual addition | accuracy | Hurt |
| route 10% of inputs to a random tokenizer (ablation) | MoTo-Reg. · synthetic multilingual addition | accuracy | Hurt |
| route all inputs randomly (ablation) | MoTo-Random · synthetic multilingual addition | accuracy | Hurt |
| use 16K per-language vocabularies instead of 8K (ablation) | MoTo · multilingual downstream tasks | accuracy | Improved |
| route code-switched Farsi inputs to the English tokenizer instead of the Farsi one (ablation) | MoTo · code-switched ToksSuite Farsi | accuracy: 70 -> 33 | Hurt |
| route Spanish code-mixed sentiment inputs to English tokenizer (ablation) | MoTo · CodeMixBench sentiment (Spanish) | accuracy: 44 -> 49 | Improved |
| use GlotLID routing instead of oracle language ID (ablation) | MoTo · code-switched ToksSuite | accuracy: 70 -> 72 | No clear effect |
| use BPE-Dropout with Llama tokenizer (ablation) | Llama · English benchmarks | accuracy relative to Llama-3: -0.33 ± 0.37 | No clear effect |
| use GPT-4 pre-tokenizer instead of language-specific pre-tokenization (ablation) | MoTo · English benchmarks | accuracy relative to Llama-3: +0.54 -> +0.04 | No clear effect |
Closest papers by meaning
- Enhancing Large Language Models through Adaptive Tokenizers (NeurIPS 2024 · similarity 0.91)
- Explaining and Mitigating Crosslingual Tokenizer Inequities (NeurIPS 2025 · similarity 0.89)
- MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization (NeurIPS 2024 · similarity 0.89)
- Data Mixture Inference Attack: BPE Tokenizers Reveal Training Data Compositions (NeurIPS 2024 · similarity 0.89)
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.89)
- The Foundations of Tokenization: Statistical and Computational Concerns (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
