Research map / Label-efficient and robust learning / Knowledge distillation
Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillation
Buu Phan, Ashish J Khisti, Karen Ullrich
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FKL distillation instead of plain SFT | Qwen2.5-1.5B-Instruct 32k vocab · GSM8K 4-shot | accuracy: 54.0 -> 58.6 | Improved |
| add SFT loss to FKL distillation | FKL, 32k vocab · GSM8K 4-shot | accuracy: 58.6 -> 63.0 | Improved |
| use FKL instead of ALM | Qwen2.5-1.5B-Instruct 64k vocab · GSM8K 4-shot | accuracy: 56.2 -> 61.8 | Improved |
| trim vocabulary to 32k with FKL+SFT vs original model | Qwen2.5-1.5B-Instruct · GSM8K 4-shot | accuracy: 58.1 -> 63.0 | Improved |
| add SFT loss to FKL distillation | FKL, 16k vocab · HumanEval 0-shot | pass@1: 42.6 -> 46.4 | Improved |
| use ALM instead of SFT | Qwen2.5-1.5B-Instruct 32k vocab · HumanEval 0-shot | pass@1: 46.9 -> 32.3 | Hurt |
| use FKL instead of SFT | Qwen2.5-1.5B-Instruct 16k vocab · MBPP 3-shot | pass@1: 30.8 -> 40.8 | Improved |
| add SFT loss to FKL distillation | FKL, 32k vocab · MBPP 3-shot | pass@1: 41.8 -> 40.6 | Hurt |
| increase trimmed vocab from 16k to 64k for FKL | Qwen2.5-1.5B-Instruct · GSM8K 4-shot | accuracy: 56.5 -> 61.8 | Improved |
| use FKL instead of SFT at full vocab | Qwen2.5-1.5B-Instruct full vocab · GSM8K 4-shot | accuracy: 60.2 -> 61.1 | Improved |
Closest papers by meaning
- Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching (NeurIPS 2025 · similarity 0.91)
- Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models (AAAI 2025 · similarity 0.91)
- Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.91)
- Enhancing Large Language Models through Adaptive Tokenizers (NeurIPS 2024 · similarity 0.89)
- WASD: Wasserstein-based Knowledge Distillation for Large Language Models (NeurIPS 2026 · similarity 0.89)
- Data Mixture Inference Attack: BPE Tokenizers Reveal Training Data Compositions (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
