Research map / Label-efficient and robust learning / Knowledge distillation
Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MultiLevelOT, a cross-tokenizer knowledge distillation method that combines token-level (holistic absolute-difference and logarithmic) and sequence-level Sinkhorn optimal transport losses on LLM logits.
Problem
Knowledge distillation between LLMs requires identical tokenizers/vocabularies, so existing KD losses cannot transfer knowledge between teacher and student from different model families.
Approach
Jointly optimizes all tokens in a sequence using a shared sequence-level ranking and top-k truncation to get token-level OT costs (absolute and log forms), plus a sequence-level Sinkhorn distance built from token-to-token OT costs, avoiding dimensional or token-by-token correspondence.
Key results
- On labeled distillation, MultiLevelOT reaches average QED F1 60.99 vs 59.30 for ULD and 58.41 for SFT across OPT-350M, Pythia-410M and Bloomz-560M students.
- In unlabeled distillation, MultiLevelOT averages 53.22 QED F1 vs 51.94 for ULD and 50.99 for raw text.
- With mT0-300M student and LLaMA teacher, MultiLevelOT gets 41.37 QED F1 vs 37.25 for ULD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MultiLevelOT instead of ULD | OPT-350M, Pythia-410M, and Bloomz-560M students · QED, labeled distillation | F1 | Helped |
| Add sequence-aware token-level and sequence-level OT distillation | Average student results · QED, FairytaleQA, and DIALOGSum, labeled distillation | F1 / Rouge-LSum: 59.30 -> 60.99; 47.83 -> 49.00; 36.17 -> 37.58 | Helped |
| Add sequence-aware token-level and sequence-level OT distillation | Average student results · QED, FairytaleQA, and DIALOGSum, unlabeled distillation | F1 / Rouge-LSum: 51.94 -> 53.22; 42.42 -> 43.28; 33.46 -> 36.83 | Helped |
| Add absolute-difference OT with token-wise ranking (ablation) | CE-only student training · QED, labeled distillation | F1: 55.71 -> 56.76; 59.03 -> 59.71; 60.48 -> 61.22 | Helped |
| Add sequence-level ranking and truncation to AD (ablation) | QED student training · QED | F1: 56.76 -> 58.01; 59.71 -> 60.22; 61.22 -> 61.58 | Helped |
| Add sequential logarithmic loss (ablation) | AD + SR + truncation · QED | F1: 58.01 -> 58.17; 60.22 -> 61.10; 61.58 -> 61.87 | Mixed |
| Add sequence-level Sinkhorn distance loss (ablation) | AD + SR + truncation + SL · QED | F1: 58.17 -> 58.97; 61.10 -> 61.79; 61.87 -> 62.58 | Helped |
| Use sequence-level rather than token-level Sinkhorn distance (ablation) | MultiLevelOT · QED | F1: 58.32 -> 58.97; 61.22 -> 61.79; 61.95 -> 62.58 | Helped |
| Increase Sinkhorn iterations from 5 to 20 (ablation) | OPT-350M and Pythia-410M · QED | F1: 58.26 -> 58.97; 60.56 -> 61.79 | Helped |
| Increase Sinkhorn iterations from 20 to 50 (ablation) | OPT-350M and Pythia-410M · QED | F1: 58.97 -> 59.02; 61.79 -> 61.76 | No clear effect |
| Set truncation threshold k to 50 rather than 5 (ablation) | OPT-350M and Pythia-410M · QED | F1: 58.54 -> 58.97; 61.42 -> 61.79 | Helped |
| Increase truncation threshold k from 50 to 1000 (ablation) | OPT-350M and Pythia-410M · QED | F1: 58.97 -> 58.42; 61.79 -> 61.32 | Hurt |
Surprising findings
- Token-level Sinkhorn distance gives only marginal gains over no Sinkhorn loss, while sequence-level Sinkhorn gives larger gains.
Limitations
- Requires hyperparameter choices (k, N, lambda, loss weights) that are set empirically.
- DSKD excluded from comparison due to added learnable projectors.
- Evaluated only on three QA/summarization tasks with small student models (up to 1B).
Experimental setup
Datasets: QED FairytaleQA DIALOGSum
Benchmarks: QED FairytaleQA DIALOGSum
Metrics: F1 Rouge-LSum
Baselines: SFT SeqKD MinED ULD Raw Text
Builds on: ULD SinKD DSKD MinED SeqKD
Closest papers by meaning
- WASD: Wasserstein-based Knowledge Distillation for Large Language Models (NeurIPS 2026 · similarity 0.92)
- Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching (NeurIPS 2025 · similarity 0.91)
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.90)
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.90)
- TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.89)
- MiniPLM: Knowledge Distillation for Pre-training Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.