Research map / Large language models / Language, safety and interpretability
Token-Scaled Logit Distillation for Ternary Weight Generative Language Models
Minsoo Kim, Sihwa Lee, Janghwan Lee, Sukjin Hong, Du-Seong Chang, Wonyong Sung, Jungwook Choi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Logit KD with token-scaled logit distillation (TSLD) | W2A16 QAT of OPT-6.7B · PTB | perplexity: 11.33 -> 11.17 | Improved |
| add ground-truth loss naively to Logit KD (Logit+GT) (ablation) | W2A16 QAT of OPT-1.3B · PTB | perplexity: 14.80 -> 16.41 | Hurt |
| add ground-truth loss naively to Logit KD (Logit+GT) (ablation) | W4A16 QAT of OPT-1.3B · PTB | perplexity: 13.73 -> 15.03 | Hurt |
| scale Logit+GT with per-token teacher cross-entropy weights (TSLD) (ablation) | W2A16 QAT of OPT-1.3B · PTB | perplexity: 16.41 -> 14.60 | Improved |
| add layer-to-layer (L2L) distillation to Logit KD (ablation) | W2A16 QAT of OPT-1.3B · PTB | perplexity: 14.80 -> 17.62 | Hurt |
| TSLD instead of Logit KD | OPT-6.7B channel-wise W2 · GSM8K | accuracy: 21.08 -> 24.49 | Improved |
| TSLD instead of Logit KD | LLaMA-7B channel-wise W2 · GSM8K | accuracy: 25.47 -> 26.23 | Improved |
| TSLD instead of Logit KD | OPT-2.7B W2A16 · ARC_challenge | accuracy: 31.91 -> 33.45 | Improved |
| TSLD instead of Logit KD | OPT-2.7B W2A16 · GSM8K | accuracy: 20.02 -> 20.24 | No clear effect |
| TSLD instead of Logit KD | OPT-1.3B W2A16 · GLUE CoLA | accuracy: 48.72 -> 54.47 | Improved |
| QAT with TSLD instead of 4-bit PTQ (OPTQ) | OPT-1.3B W4A16 · GLUE RTE | accuracy: 56.32 -> 75.97 | Improved |
| channel-wise instead of tensor-wise quantization (ablation) | TSLD W2A16 on GPT-Neo-1.3B · PTB | perplexity: 30.60 -> 19.27 | Improved |
Closest papers by meaning
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (ICLR 2024 · similarity 0.87)
- Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models (AAAI 2025 · similarity 0.87)
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.87)
- Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation (NeurIPS 2025 · similarity 0.86)
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.86)
- QT-DoG: Quantization-Aware Training for Domain Generalization (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
