Research map / Large language models / Language, safety and interpretability

Model-Aware Tokenizer Transfer

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MATT, which transfers a pretrained LLM to a new tokenizer by distilling inter-token attention influence (AIM) from the original model into the new-tokenizer model, training only input embeddings as a warm-up before LM training.

Problem

Tokenizer transfer for adapting pretrained LLMs to new languages with better-compressing tokenizers requires initializing new embeddings well; existing heuristic methods ignore higher-layer model dynamics and gradient-based NTP training is computationally expensive.

Approach

Attention Influence Modeling aligns segment-level weighted value states of attention layers between the original-tokenizer teacher and the new-tokenizer student (using offset-based segmentation), training only the new input embeddings over a few layers, avoiding full-model language modeling passes.

Key results

• On Gemma 3 12B PT (Ukrainian), MATT averages 77.27 on discriminative tasks vs 78.18 for the original model, and 11.81 vs 12.68 on generative tasks, beating FOCUS and NTP baselines.
• Across Gemma 3 4B and Qwen 3 4B on Arabic, German, Japanese, Swahili, MATT beats FOCUS and Token Distillation on every aggregate metric (e.g., Gemma 3 4B Avg Disc 66.03 vs 57.39 for Token Distillation at similar compute).
• MATT with 7h training on Gemma 3 12B matches or beats NTP trained with 10.5h, using only about 240M tokens.

Surprising findings

• Applying AIM to all layers up to depth n performed worse than applying it only to the final layer n, both in efficiency and downstream accuracy.
• Token Distillation collapsed to near-zero BLEU on en-to-uk translation despite near-original uk-to-en quality on Gemma 3 12B.

Limitations

• Only input embeddings are optimized; generative gap remains and untied-embedding models need an added NTP loss.
• Self-distillation upper-bounded by the teacher; cannot add capabilities for languages the base model lacks (e.g., Swahili on Qwen 3 4B).
• Requires an extra forward pass through the original model; evaluation limited to few languages and no full continual pretraining experiments.

Experimental setup

Datasets: Kobza HPLT 2.0 Cleaned OpenSubtitles NLLB

Benchmarks: Belebele Global MMLU MMMLU Long FLORES WMT24++ XL-Sum

Metrics: accuracy BLEU

Baselines: WECHSEL Transtokenizers TokAlign FOCUS NTP Token Distillation

Builds on: FOCUS Token Distillation WECHSEL TokAlign Transtokenization

Closest papers by meaning

• Enhancing Large Language Models through Adaptive Tokenizers (NeurIPS 2024 · similarity 0.92)
• Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.90)
• Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.90)
• Zero-Shot Tokenizer Transfer (NeurIPS 2024 · similarity 0.90)
• Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching (NeurIPS 2025 · similarity 0.88)
• The Foundations of Tokenization: Statistical and Computational Concerns (ICLR 2025 · similarity 0.88)