AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

Gül Sena Altıntaş, Malikeh Ehghaghi, Brian Lester, Fengyuan Liu, Wanru Zhao, Marco Ciccone, Colin Raffel

ICML 2026 · Large language models › Language, safety and interpretability · approach: Controlled text generation · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use TokenMonster tokenizer (ungreedy, consistency constraints) instead of larger multilingual tokenizers1B LM · TokSuite multilingual perturbationsrelative accuracy drop: 0.17 vs 0.22 (XGLM, BLOOM, ByT5) and 0.26-0.27 (Aya, Tekken)Improved
use byte-level ByT5 tokenizer1B LM · TokSuite noise perturbations (non-English)relative accuracy drop: 0.18Improved
use byte-level ByT5 tokenizer1B LM · Flores200 efficiencysubword fertility: 5.79 vs 1.54 (mBERT)Hurt
apply noise perturbations in non-English instead of Englishall 14 tokenizers · TokSuite noiserelative accuracy drop: 0.15 -> 0.21Hurt
apply Unicode styling perturbationsall 14 tokenizers · TokSuite Unicode (EN)relative accuracy drop: 0.53Hurt
add NFKC normalization (XGLM)XGLM · TokSuite Unicode (EN)relative accuracy drop: 0.11 vs 0.53 averageImproved
scale model from 300M to 1B to 7B with fixed tokenizerLlama-3.2 · TokSuite perturbations (average)relative accuracy drop: 0.26, 0.26, 0.22No clear effect
increase tokenizer vocabulary sizetokenizers · TokSuite robustnessrelative accuracy dropNo clear effect
use multilingual-specialized tokenizer (XGLM)XGLM · Flores200parity: 1.18 averageImproved
use mBERT tokenizermBERT · Flores200subword fertility: 1.54Improved
train industry-scale models much longerlarger models · TokSuiterobustnessMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy