Research map / Label-efficient and robust learning / Knowledge distillation
Generalized Kullback–Leibler Divergence Loss
Jiequan Cui, Beier Zhu, Qingshan Xu, Zhuotao Tian, Xiaojuan Qi, Bei Yu, Hanwang Zhang, Richang Hong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| break asymmetric optimization (BA) by enabling gradient on wMSE (ablation) | DKL loss · CIFAR-100 AT / ImageNet KD | AA / top-1: 30.20 -> 30.42 AA; 71.03 -> 71.60 top-1 | Improved |
| smoother sample-wise weight function (gamma 0.4) (ablation) | DKL with BA · CIFAR-100 AT | clean / AA: 62.69 -> 64.15 clean; 30.42 -> 31.70 AA | Improved |
| class-wise weight function instead of sample-wise (ablation) | DKL with BA · CIFAR-100 AT | clean / AA: 64.15 -> 65.76 clean; 31.70 -> 31.91 AA | Improved |
| class-wise smoother weight function (ablation) | DKL with BA · ImageNet KD | top-1: 71.80 -> 71.91 | Improved |
| replace KL with DKL (equivalent form) (ablation) | KL baseline · CIFAR-100 AT | AA: 30.29 -> 30.20 | No clear effect |
| GKL instead of KL, temperature 1.0 | KL-KD · ImageNet-LT | All accuracy: 44.32 -> 45.40 | Improved |
| raise KD temperature from 1.0 to 2.0 (ablation) | KL-KD · ImageNet-LT | Many / Few accuracy: 64.60 -> 66.50 Many; 9.53 -> 6.54 Few | Mixed |
| GKL-KD instead of KL-KD (ResNeXt-101 to ResNet-50) | KL-KD · ImageNet-LT | Many / Medium accuracy: 68.83 -> 70.31 Many; 42.31 -> 43.47 Medium | Improved |
| class-wise global information with gamma 0.3 (ablation) | KD weight function · ImageNet-LT | accuracy: 44.32 -> 45.40 | Improved |
| larger KD temperature t=1 to t=8 (ablation) | KL-KD CLIP · ImageNet zero-shot | top-1: 57.28 -> 60.85 | Improved |
| GKL loss instead of KL | FixMatch semi-supervised · CIFAR-100 | top-1: 69.09 -> 70.00 | Improved |
| GKL loss instead of MSE consistency | Mean-Teacher · CIFAR-100 | top-1: 67.38 -> 70.05 | Improved |
Closest papers by meaning
- Decoupled Kullback-Leibler Divergence Loss (NeurIPS 2024 · similarity 0.98)
- Statistical and Geometrical properties of the Kernel Kullback-Leibler divergence (NeurIPS 2024 · similarity 0.89)
- Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss (ICML 2025 · similarity 0.86)
- Loss Functions and Operators Generated by f-Divergences (ICML 2025 · similarity 0.86)
- Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation (NeurIPS 2024 · similarity 0.86)
- Better Estimation of the Kullback--Leibler Divergence Between Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
