Research map / Large language models / Language, safety and interpretability
Language Model Tokenizers Introduce Unfairness Between Languages
Aleksandar Petrov, Emanuele La Malfa, Philip Torr, Adel Bibi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Switch from English-centric r50k tokenizer to multilingual XLM-R tokenizer | tokenizer · FLORES-200 Shan | premium vs English: 18.76 -> 4.43 | Improved |
| Switch from r50k_base to larger-vocabulary cl100k_base (ChatGPT/GPT-4) | tokenizer · FLORES-200 Shan | premium vs English: 18.76 -> 15.05 | Improved |
| Switch from r50k_base to cl100k_base | tokenizer · FLORES-200 German | premium vs English: 2.14 -> 1.58 | Improved |
| Use byte-level BLOOM tokenizer designed for equal token ratios | tokenizer · FLORES-200 Santali | premium vs English: 12.71 | No clear effect |
| Use NLLB tokenizer (trained on all 200 languages) instead of BLOOM | tokenizer · FLORES-200 Santali | premium vs English: 12.71 -> 2.49 | Improved |
| Use UTF-8 bytes (ByT5) instead of UTF-32 codepoints (CANINE) | byte-level tokenization · FLORES-200 Chinese (Simplified) | encoding length premium: 0.34 -> 0.93 | Hurt |
| Use UTF-8 bytes (ByT5) instead of UTF-32 codepoints (CANINE) | byte-level tokenization · FLORES-200 Shan | encoding length premium: 1.42 -> 3.94 | Hurt |
| Use language-targeted tokenizer GottBERT instead of English-centric RoBERTa for German speakers | tokenizer · FLORES-200 English vs Dutch | premium: 1.35 vs 1.73 | No clear effect |
| Use CamemBERT French tokenizer | tokenizer · FLORES-200 English vs Catalan | premium: 1.20 vs 1.59 | No clear effect |
| Use MuRIL tokenizer targeting Indian languages | tokenizer · FLORES-200 Kashmiri vs English | premium: 1.00 vs 1.75 | No clear effect |
| Use FlanT5 tokenizer with UNK restriction | tokenizer · FLORES-200 coverage | premium range: 1.37-3.41 | Mixed |
| Reduce vocabulary to one third for ChatGPT/GPT-4 tokenizer | cl100k_base tokenizer · English sequence length | token length: 10% longer | Hurt |
Closest papers by meaning
- Explaining and Mitigating Crosslingual Tokenizer Inequities (NeurIPS 2025 · similarity 0.89)
- MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization (NeurIPS 2024 · similarity 0.87)
- From Language Models over Tokens to Language Models over Characters (ICML 2025 · similarity 0.86)
- Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.85)
- Model-Aware Tokenizer Transfer (NeurIPS 2026 · similarity 0.85)
- The Foundations of Tokenization: Statistical and Computational Concerns (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
