Research map / Large language models / Language, safety and interpretability
Zero-Shot Tokenizer Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces zero-shot tokenizer transfer (ZeTT) and a hypernetwork that predicts embeddings for an arbitrary tokenizer, enabling LMs (XLM-R, Mistral-7B) to switch tokenizers with little loss and fast recovery via <1B tokens of continued training.
Problem
Language models are bound to the tokenizer they were trained with; swapping in a new tokenizer (e.g., more efficient for other languages or code) requires embeddings for new tokens, and prior heuristics for initializing them perform near chance in a zero-shot setting.
Approach
Train a hypernetwork that takes a tokenizer's vocabulary (each token decomposed via the original tokenizer and embedded with the original embeddings) and predicts input/output embeddings, trained over a diverse distribution of sampled UnigramLM tokenizers so it generalizes to unseen tokenizers without training on them.
Key results
- On XNLI with XLM-R, the hypernetwork preserves average accuracy to within 1% (71.2 vs 72.6 original) with 14% shorter sequences, outperforming FOCUS (68.6).
- Zero-shot Mistral-7B transferred to GPT2 tokenizer averages 73.0 on English benchmarks versus 55.6 for FOCUS and 76.9 original; continued training on 800M tokens brings it to 74.9.
- On MT-Bench, zero-shot transfer of Mistral-7B-Instruct-v0.1 scores 6.56 vs 7.33 original and 5.03 for FOCUS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the original XLM-R tokenizer with a language-specific tokenizer and use ZeTT hypernetwork embeddings | XLM-R with reused English XNLI adapters · XNLI (13-language average) | accuracy: 72.6 -> 71.2; -1% | Hurt |
| Replace the original tokenizer with language-specific tokenizers | XLM-R with reused English XNLI adapters · XNLI (13-language average) | token length: -14% | Helped |
| Use ZeTT hypernetwork instead of FOCUS for zero-shot tokenizer transfer | Mistral-7B · English benchmarks (Avg., GPT2 tokenizer) | accuracy: 55.6 -> 73.0 | Helped |
| Use ZeTT hypernetwork instead of FOCUS for zero-shot tokenizer transfer | Mistral-7B · HumanEvalPack (Avg. pass@1, StarCoder tokenizer) | pass@1: 13.3 -> 23.2 | Helped |
| Continue training with the StarCoder tokenizer for 800M tokens rather than continue with the original tokenizer | Mistral-7B · HumanEvalPack (Avg. pass@1) | pass@1: 26.7 -> 27.6 | Helped |
| Replace the original tokenizer with language-specific tokenizers and use ZeTT hypernetwork embeddings | Mistral-7B · XCOPA (Avg.) | accuracy: 54.6 -> 57.1; +3% | Helped |
| Replace the original tokenizer with language-specific tokenizers | Mistral-7B · XCOPA (Avg.) | token length: -54% | Helped |
| Use ZeTT hypernetwork instead of FOCUS for zero-shot transfer | Mistral-7B · Multilingual MMLU (5-shot, German) | accuracy: 26.2 -> 43.7 | Helped |
| Replace the original tokenizer with a language-specific tokenizer and use ZeTT hypernetwork embeddings | Mistral-7B · Multilingual MMLU (5-shot, German) | accuracy: 51.6 -> 43.7; -8% | Hurt |
| Add inter-token attention (ablation) | ZeTT hypernetwork · Sampled tokenizers | bits-per-byte: 1.157 -> 1.118 | Helped |
| Add inter-token attention (ablation) | ZeTT hypernetwork · GPT-NeoX tokenizer | bits-per-byte: 0.902 -> 0.904 | Hurt |
| Add inter-token attention (ablation) | ZeTT hypernetwork · English UnigramLM tokenizer | bits-per-byte: 1.054 -> 1.103 | Hurt |
Surprising findings
- Copying embeddings of exactly overlapping tokens is not an optimal initialization; GPT2 original embeddings degrade significantly when switched to a frequency-based UnigramLM tokenization function.
- Continued training with the original tokenizer slightly degrades Mistral-7B average performance, while continued training with the new StarCoder tokenizer performs better.
Limitations
- A hypernetwork must be trained separately for every base model, which is computationally intensive.
- Assumes fixed pretokenization and UnigramLM parametrization of tokenization function; the tokenization function is not an input to the hypernetwork.
- Experiments limited to text-only models.
Experimental setup
Datasets: MADLAD-400 StarCoder data Flan v2 XNLI SICK
Benchmarks: PiQA HellaSwag ARC BoolQ MMLU HumanEvalPack XNLI XCOPA MT-Bench Farsi NLI Dutch NLI Aymara NLI
Metrics: accuracy pass@1 score (1 to 10) bits-per-character relative sequence length change
Baselines: FOCUS FVT OFA Lexical RAMEN WECHSEL
Builds on: FOCUS MIMICK FVT WECHSEL Task Arithmetic
Closest papers by meaning
- Model-Aware Tokenizer Transfer (NeurIPS 2026 · similarity 0.90)
- Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching (NeurIPS 2025 · similarity 0.88)
- LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings (NeurIPS 2024 · similarity 0.87)
- Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.87)
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.87)
- Enhancing Large Language Models through Adaptive Tokenizers (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.