Research map / Large language models / Language, safety and interpretability
Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Approximate Likelihood Matching (ALM), a cross-tokenizer distillation objective that matches binarised chunk-level likelihoods of aligned token spans, and evaluates it on tokenizer transfer, large-to-small distillation, and hypernetwork zero-shot tokenizer transfer.
Problem
Existing distillation methods require equivalent or similar tokenizers between teacher and student, which limits teacher-student pairs; prior cross-tokenizer heuristics are also hybrid rather than pure distillation.
Approach
Align teacher and student token chunks that encode the same text, then minimise an f-divergence between their chunk-level likelihoods binarised as {p, 1-p}, with outcome-chunk debiasing to remove tokenization bias and optional hidden-state alignment; this works across fundamentally different tokenizers including byte-level.
Key results
- On Gemma2 2B IT transfer to Qwen2 tokenizer, ALM reaches 55.1 average vs 53.0 for MinED and 51.6 for SFT.
- Distilling OpenMath2-Llama3.1-8B into Gemma2 2B, ALM+SFT reaches 70.2 GSM8K and 53.3 average, up to ~3 points above SFT.
- ALM hypernetwork training improves zero-shot transfer to Mistral v2 BoolQ from 67.5 (SFT) to 74.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ALM instead of MinED for transfer to the Qwen2 tokenizer | Gemma2 2B IT · Qwen2 tokenizer transfer | Avg.: 53.0 -> 55.1 | Helped |
| use ALM instead of MinED for transfer to the Qwen2 tokenizer | Llama3.2 3B IT · Qwen2 tokenizer transfer | Avg.: 57.3 -> 58.6 | Helped |
| use ALM + SFT instead of SFT for byte transfer | Gemma2 2B IT · byte tokenizer transfer | Avg.: 46.5 -> 51.3 | Helped |
| use ALM + SFT instead of SFT for byte transfer | Llama3.2 3B IT · byte tokenizer transfer | Avg.: 52.6 -> 53.5 | Helped |
| add next-token prediction objective to ALM | Gemma2 2B IT · Qwen2 tokenizer transfer | Avg.: 55.1 -> 54.9 | Hurt |
| add next-token prediction objective to ALM | Gemma2 2B IT · byte tokenizer transfer | Avg.: 50.6 -> 51.3 | Helped |
| replace SFT with ALM + SFT | Gemma2 2B · GSM8K (zero-shot CoT) | accuracy: 67.2 -> 70.2 | Helped |
| replace SFT hypernetwork objective with ALM | Gemma2 2B hypernetwork · BoolQ, zero-shot transfer to Mistral v2 tokenizer | accuracy: 67.5 -> 74.0 | Helped |
| add outcome chunk debiasing (ablation) | ALM for Gemma2 2B IT · Qwen2 tokenizer transfer | Avg.: 54.1 -> 55.5 | Helped |
| add a nonzero bias threshold (γ = 0.10) (ablation) | ALM with outcome debiasing for Gemma2 2B IT · Qwen2 tokenizer transfer | Avg.: 55.5 -> 55.6 | Helped |
| add hidden-state distillation loss (ablation) | ALM with outcome debiasing for Gemma2 2B IT · Qwen2 tokenizer transfer | Avg.: 55.5 -> 55.0 | Hurt |
| add hidden-state distillation loss (ablation) | ALM with outcome debiasing for Llama3 · byte tokenizer transfer | Avg.: 45.3 -> 47.0 | Helped |
Surprising findings
- Plain SFT is competitive with prior cross-tokenizer distillation methods (DSKD, MinED) in the large-to-small math distillation setting.
- Hidden-state loss is slightly detrimental for subword-to-subword transfer but highly beneficial for subword-to-byte transfer.
Limitations
- Trained on only about 0.6B tokens due to compute constraints, so behaviour at larger token budgets is unclear.
- Explores only one instantiation (binarised KL/TVD) and a greedy, non-overlapping chunk selection strategy.
- Restricted to supervised KD; on-policy or rejection-sampling distillation not explored.
Experimental setup
Datasets: Tulu3 instruction-tuning dataset OpenMathInstruct-2
Benchmarks: PiQA ARC-Challenge BoolQ MMLU AGIEval IFEval GSM8K MATH ARC-E HellaSwag
Metrics: accuracy zero-shot accuracy
Baselines: SFT DSKD MinED ULD ZeTT SFT hypernetwork DeePEn
Builds on: ULD MinED DSKD GradNorm ZeTT
Closest papers by meaning
- Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models (AAAI 2025 · similarity 0.91)
- A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone (NeurIPS 2025 · similarity 0.89)
- Zero-Shot Tokenizer Transfer (NeurIPS 2024 · similarity 0.88)
- Model-Aware Tokenizer Transfer (NeurIPS 2026 · similarity 0.88)
- WASD: Wasserstein-based Knowledge Distillation for Large Language Models (NeurIPS 2026 · similarity 0.88)
- SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.