AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

MoTo: Mixture of Tokenizers Towards Fair Multilingual Language Modeling

Gül Sena Altıntaş, Colin Raffel

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Mixture of Tokenizers (MoTo), a modular tokenization approach for multilingual language models aimed at fairer treatment of morphologically rich and non-Latin-script languages.

Problem

Single, English-dominant tokenizers fixed at training time fragment morphologically rich and non-Latin-script languages, producing longer sequences and lower information density, which hurts multilingual fairness.

Approach

A modular mixture of tokenizers that combines multiple tokenizers rather than relying on one fixed tokenizer chosen at training time; abstract is truncated so specifics are unknown.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use language-specific per-language tokenizers (8K each) instead of one Llama-3 tokenizerMoTo · English benchmarks, 1B modelaccuracy relative to Llama-3: +0.54 ± 0.38No clear effect
use per-language tokenizers instead of Llama tokenizerMoTo · multilingual downstream tasks, 20 non-English languagesaccuracyImproved
use jointly trained mBPE instead of per-language tokenizers (ablation)mBPE (2K/42K) · synthetic multilingual additionaccuracyHurt
route 10% of inputs to a random tokenizer (ablation)MoTo-Reg. · synthetic multilingual additionaccuracyHurt
route all inputs randomly (ablation)MoTo-Random · synthetic multilingual additionaccuracyHurt
use 16K per-language vocabularies instead of 8K (ablation)MoTo · multilingual downstream tasksaccuracyImproved
route code-switched Farsi inputs to the English tokenizer instead of the Farsi one (ablation)MoTo · code-switched ToksSuite Farsiaccuracy: 70 -> 33Hurt
route Spanish code-mixed sentiment inputs to English tokenizer (ablation)MoTo · CodeMixBench sentiment (Spanish)accuracy: 44 -> 49Improved
use GlotLID routing instead of oracle language ID (ablation)MoTo · code-switched ToksSuiteaccuracy: 70 -> 72No clear effect
use BPE-Dropout with Llama tokenizer (ablation)Llama · English benchmarksaccuracy relative to Llama-3: -0.33 ± 0.37No clear effect
use GPT-4 pre-tokenizer instead of language-specific pre-tokenization (ablation)MoTo · English benchmarksaccuracy relative to Llama-3: +0.54 -> +0.04No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy