Research map / Large language models / Language, safety and interpretability
Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VocAgnoLM, which aligns student and teacher tokens across different vocabularies via character offsets and reweights student tokens by teacher token loss, enabling continual pretraining of TinyLlama 1.1B with 7B math teachers that have different vocabularies.
Problem
Knowledge-guided language modeling assumes teacher and student share a vocabulary; vocabulary mismatch causes divergent token sequences and logit distributions, which blocks using stronger teachers (e.g., Qwen2.5-Math) for a small student.
Approach
Token-level Lexical Alignment maps each student token to the minimal range of teacher tokens covering its character span (one-to-many, via binary search), and a Teacher Guided Loss selects student tokens by the difference between student and aggregated mapped-teacher token losses instead of matching logit distributions.
Key results
- With Qwen2.5-Math-Instruct as teacher (6.32% vocabulary overlap), VocAgnoLM reaches AVG 29.9 vs 22.4 for ULD and about 46% over naive TinyLlama-CPT (20.5).
- Using Llemma (same vocabulary) VocAgnoLM achieves AVG 24.1 vs 21.4 for KLD.
- Token-level Lexical Alignment reaches 100% student-token coverage (IoS) and outperforms 8-64 chunk alignment (best chunked 18.9 AVG).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VocAgnoLM instead of ULD with Qwen2.5-Math-Instruct as teacher | TinyLlama 1.1B continual pretraining · Math evaluation suite | AVG: 22.4 -> 29.9 | Helped |
| use VocAgnoLM instead of ULD with Mistral-ProXMath as teacher | TinyLlama 1.1B continual pretraining · Math evaluation suite | AVG: 21.7 -> 24.2 | Helped |
| use VocAgnoLM instead of KLD with Llemma as teacher | TinyLlama 1.1B continual pretraining · Math evaluation suite | AVG: 21.4 -> 24.1 | Helped |
| replace token-level lexical alignment with chunk alignment (ablation) | Teacher-guided training on 5B OpenWebMath tokens · Math evaluation suite | AVG: 20.2 -> 18.9 | Hurt |
| include unmapped student tokens rather than exclude them (ablation) | Teacher-guided training on 2B tokens · Math evaluation suite | AVG: 6.8 -> 16.6 | Helped |
| include unmapped student tokens rather than use batch Mean guidance (ablation) | Teacher-guided training on 2B tokens · Math evaluation suite | AVG: 16.3 -> 16.6 | No clear effect |
| aggregate multi-mapped teacher token losses with Max instead of Mean (ablation) | Teacher-guided training on 2B tokens · Math evaluation suite | AVG: 16.6 -> 16.5 | No clear effect |
| aggregate multi-mapped teacher token losses with Max instead of Mean (ablation) | Teacher-guided training on 15B tokens · Math evaluation suite | AVG: 24.3 -> 24.2 | No clear effect |
| use a stronger teacher model | VocAgnoLM · Math evaluation suite | AVG: 24.1 -> 24.2; 24.1 -> 25.9; 26.9 -> 29.9 | Mixed |
Surprising findings
- Chunk-based alignment with 64 chunks performs worse than random chunk selection, showing that coarse alignment can harm guidance.
- Excluding unmapped student tokens (e.g., special tokens) severely degrades performance (AVG 6.8 vs 16.6 when included at 2B tokens).
Limitations
- Evaluated only on one student (TinyLlama 1.1B) and one domain corpus (OpenWebMath, math), as acknowledged by the authors.
- Relies on the Llemma and other teachers being used in continual pretraining; large-scale validation left to future work.
- Results on some benchmarks (e.g., SAT with 32 questions) are noisy and reported with exclusions.
Experimental setup
Datasets: OpenWebMath
Benchmarks: GSM8K MATH SVAMP ASDiv MAWPS TabMWP MathQA MMLU-STEM SAT
Metrics: accuracy average accuracy
Baselines: KLD ULD Rho-1 TinyLlama-CPT
Builds on: Rho-1 Universal Logit Distillation LitGPT TinyLlama
Closest papers by meaning
- MiniPLM: Knowledge Distillation for Pre-training Language Models (ICLR 2025 · similarity 0.86)
- Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training (NeurIPS 2025 · similarity 0.86)
- DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs (ICML 2025 · similarity 0.86)
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.85)
- Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models (NeurIPS 2024 · similarity 0.85)
- Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.