Research map / Label-efficient and robust learning / Knowledge distillation
Bridging the Tokenizer Gap: Semantics and Distribution-aware Knowledge Transfer for Unbiased Cross-Tokenizer Distillation
Huazheng Wang, Yongcheng Jing, Haifeng Sun, Jingyu Wang, Jianxin Liao, Leszek Rutkowski, Dacheng Tao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SEDI instead of the best cross-tokenizer baseline DSKD | student distillation · Dolly (LLaMA2-7B -> GPT2-124M) | ROUGE-L: 23.33 -> 24.38 | Improved |
| use SEDI instead of DSKD on unseen UnNI | student distillation · UnNI (Dolly-Pythia-3B -> OPT-125M) | ROUGE-L: 17.36 -> 18.51 | Improved |
| use SEDI instead of DSKD on HumanEval | Pythia-160M student · HumanEval | Pass@1: 56.10 -> 60.98 | Improved |
| use SEDI instead of DSKD on HumanEval ROUGE-L | Pythia-160M student · HumanEval | ROUGE-L: 23.01 -> 22.93 | No clear effect |
| use SEDI instead of DSKD on ORCA | GPT2-Medium-355M student · ORCA | Pass@1: 2.93 -> 3.51 | Improved |
| bipartite graph alignment instead of edit-distance alignment | token alignment · Dolly | Top-1 accuracy (%): 84.94 -> 95.27 | Improved |
| increase top-K from 10 to 100 (ablation) | SEDI · S-NI | ROUGE-L: 14.97 -> 18.18 | Improved |
| increase top-K from 100 to 400 (ablation) | SEDI · UnNI | ROUGE-L: 18.51 -> 18.57 | No clear effect |
| use teacher with larger vocabulary (DeepSeek-R1-Distill) vs MetaMath-7B | student distillation · MetaMath-trained students | Pass@1 vs ROUGE-L: GSM8K Pass@1 7.65 vs 3.94 | Mixed |
| add entropy alignment with student logits (SEDI) | student distillation · S-NI/UnNI OPT-125M | ExAccErr | Improved |
| use SEDI instead of baselines for fluency | LLaMA-GPT pair · Dolly | Fluency: 4.81 -> 4.91 | No clear effect |
| use SEDI instead of DSKD training cost | LLaMA-GPT pair · Dolly | training time per step (s): 1.5276 -> 1.8432 | Hurt |
Closest papers by meaning
- Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models (AAAI 2025 · similarity 0.91)
- Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching (NeurIPS 2025 · similarity 0.90)
- WASD: Wasserstein-based Knowledge Distillation for Large Language Models (NeurIPS 2026 · similarity 0.89)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.87)
- SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation (NeurIPS 2026 · similarity 0.87)
- A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
