Research map / Large language models / Language, safety and interpretability

Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Controlled text generation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows instruction-tuned LMs are robust to random and character-level non-canonical tokenizations of their input (retaining ~93% and ~91% of performance for Qwen-2.5-7B-Instruct), that character-level and right-aligned digit grouping can improve some tasks at inference time, and that robustness arises during SFT via separation of instruction and response turns.

Problem

Deterministic BPE tokenizers map each string to one canonical token sequence, but LMs trained this way may be unable to handle other valid tokenizations of the same text at inference time; it is unclear how robust they are and where that robustness comes from.

Approach

Systematically evaluate instruction-tuned LMs on non-canonical tokenizations (uniform random segmentation and character-level) unseen in training across 20 benchmarks, use these tokenizations at inference time for performance gains on orthography-sensitive tasks, and trace the robustness to SFT turn structure and to models perceiving non-canonical text as misspellings.

Key results

• Qwen-2.5-7B-Instruct retains 93.4% of its average overall performance under random non-canonical tokenization and 90.8% under character-level tokenization across 20 benchmarks.
• Llama-3.1-8B-Instruct with right-to-left digit grouping improves 10-digit arithmetic from 36.5 to 70.2 (+33.7), and character-level tokenization improves code description from 68.6 to 82.9 (+14.3).
• Base models of OLMo-2 and Llama-3.1 score at best 0.317 on spelling and 0.260 on grammaticality under character-level input, while SFT alone yields robustness (spelling 0.786 for the no-ablation SFT model).

Measured effects reported in the paper

Surprising findings

• Instruction-tuned models retain most performance under character-level tokenization, despite never seeing such tokenizations in training.
• Base models recognize words in non-canonical form (Word Repeat >90%) yet perform at chance at identifying misspellings, suggesting they perceive non-canonical text as misspelled and mimic it.

Limitations

• Robustness is weaker for OLMo-2 (62.0% retention under character-level tokenization) and for some benchmarks such as Winograd and COPA.
• Inference-time tokenization gains are shown only on constructed tasks, and optimal tokenization is not automatically identified.
• Ablations use a Llama-3.2-1B model and a single SFT dataset.

Experimental setup

Datasets: ARC-C ARC-E COPA Winogrande Winograd CSQA OpenbookQA PIQA MMLU BoolQ HellaSwag WikidataQA

Benchmarks: ARC-C ARC-E COPA Winogrande Winograd CSQA OpenbookQA PIQA MMLU BoolQ HellaSwag WikidataQA

Metrics: accuracy retention relative to canonical win rate spelling score grammaticality score Kendall's tau

Baselines: canonical BPE tokenization

Builds on: BPE BPE-dropout AlpacaEval Tulu 3

Closest papers by meaning

• Language Models over Canonical Byte-Pair Encodings (ICML 2025 · similarity 0.91)
• From Language Models over Tokens to Language Models over Characters (ICML 2025 · similarity 0.89)
• Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.89)
• The Foundations of Tokenization: Statistical and Computational Concerns (ICLR 2025 · similarity 0.88)
• Getting the most out of your tokenizer for pre-training and domain adaptation (ICML 2024 · similarity 0.88)
• Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.88)