Research map / Architectures and efficiency / Attention and transformers
Explaining and Mitigating Crosslingual Tokenizer Inequities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains ~2,700+ comparable monolingual BPE/Unigram tokenizers for 97 languages to show token premiums persist without multilingual confounds, tests explanatory factors, and mitigates them via per-language vocabulary sizes and SuperBPE.
Problem
Token premiums (variation in tokens needed to encode parallel text across languages) are documented only in multilingual tokenizers, where data allocation confounds the effect; it is unclear what crosslingual features cause them and how to mitigate them.
Approach
Train large suites of comparably configured monolingual tokenizers across 97 languages, varying algorithm, vocabulary size, and data scaling, to isolate language-intrinsic drivers (whitespace pre-tokenization, token length, data similarity) and show that per-language optimal vocabulary sizes and superword (SuperBPE) tokenizers reduce the disparities.
Key results
- Parallel-data tokenizers reduce CTC only ~1% on average (paired t-test, p=0.0197), so training data similarity does not meaningfully explain token premiums.
- Per-language optimal vocabulary sizes significantly reduce CTC variance (F-test F=0.150, p<0.001) relative to same-vocabulary BPE tokenizers.
- SuperBPE tokenizers show lower average CTCs and less variance at every vocabulary size above 64000.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BPE instead of Unigram | Monolingual tokenizers · FLORES-200 | CTC | Helped |
| Use SentencePiece Unigram instead of BPE | Monolingual tokenizers · FLORES-200 | CTC | Hurt |
| Scale tokenizer training data by byte premium (ablation) | Monolingual BPE and Unigram tokenizers · FLORES-200 | CTC: t(3544)=-0.615, p=0.539 | No clear effect |
| Train BPE on parallel data instead of monolingual data (ablation) | BPE tokenizers for seven selected languages · FLORES-200 | token premium effects: t(152)=--2.356, p=0.0197; on average about 1% of the total CTC | No clear effect |
| Increase vocabulary size while keeping it the same across languages (ablation) | BPE tokenizers · FLORES-200 | CTC variance: F_{96,96}=1.125, p=0.565 | No clear effect |
| Set a language-specific optimal vocabulary size (ablation) | BPE tokenizers · FLORES-200 | CTC variance: F_{80, 387}=0.150, p<0.001 | Helped |
| Allow merges across whitespace boundaries (ablation) | BPE tokenizers (SuperBPE) · FLORES-200 | CTC | Helped |
| Set predicted language-specific optimal vocabulary sizes (ablation) | SuperBPE tokenizers · FLORES-200 | CTC variance: F_{558, 76}=3.8727, p<0.001 | Helped |
| Use a larger vocabulary size, identical across languages (ablation) | BPE tokenizers · FLORES-200 | CTC | Helped |
Surprising findings
- Increasing vocabulary size does not significantly reduce variance in token premiums across languages (F-test p=0.565).
- BPE tokenizers do not show a stronger byte premium to token premium relationship than Unigram; the byte-token premium relation is slightly negative.
Limitations
- Tokenizers trained on only 300MB, possibly too small especially for SuperBPE.
- Evaluation only on FLORES, which is translated from English and may have translation artifacts.
- No language models trained, so no link to downstream performance is established.
Experimental setup
Datasets: curse-of-multilinguality web text corpora NLLB parallel data FLORES-200
Benchmarks: FLORES-200
Metrics: corpus token count token premium R^2 F-test variance paired t-test
Baselines: Goldfish SentencePiece Unigram tokenizers OLMo tokenizer Pythia tokenizer SmolLM tokenizer
Builds on: SuperBPE SentencePiece Goldfish models FLORES-200
Closest papers by meaning
- Data Mixture Inference Attack: BPE Tokenizers Reveal Training Data Compositions (NeurIPS 2024 · similarity 0.89)
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.88)
- Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training (NeurIPS 2025 · similarity 0.88)
- MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization (NeurIPS 2024 · similarity 0.88)
- Enhancing Large Language Models through Adaptive Tokenizers (NeurIPS 2024 · similarity 0.88)
- A Partition Cover Approach to Tokenization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.