Research map / Label-efficient and robust learning / Knowledge distillation
Decoupled Kullback-Leibler Divergence Loss
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows KL divergence equals a decoupled loss (weighted MSE plus soft-label cross-entropy) and proposes IKL, which breaks the asymmetric gradient and adds class-wise global weights, achieving SOTA AutoAttack robustness on CIFAR and competitive KD results.
Problem
KL divergence loss is widely used in adversarial training and knowledge distillation, but its gradient mechanism is poorly understood and its asymmetric optimization can neglect part of the loss when one branch is detached (as in KD).
Approach
Prove KL gives the same gradients as a decoupled DKL = weighted MSE on logit differences (weights from softmax products) + soft-label cross-entropy; then enable gradients on the detached branch in wMSE and replace sample-wise weights with class-wise mean probability weights, yielding IKL.
Key results
- IKL-AT with WRN-34-10 on CIFAR-100 reaches 31.91% AutoAttack robustness (vs TRADES 30.29%) and 32.53% with AutoAugment, a new RobustBench SOTA without extra data.
- With 50M generated images, IKL-AT (WRN-28-10) achieves 73.85% clean and 39.18% AA on CIFAR-100, beating DM-AT's 38.83%.
- On ImageNet, IKL-KD with ResNet34 teacher gives ResNet18 71.91% top-1 vs DKD 71.70% and KD 71.03%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| enable breaking asymmetric optimization (BA) (ablation) | DKL for ImageNet knowledge distillation · ImageNet, ResNet34 teacher to ResNet18 student | Top-1 accuracy (%): 71.03 -> 71.80 | Helped |
| add class-wise global information (GI) to DKL with BA (ablation) | DKL for CIFAR-100 adversarial training · CIFAR-100, AutoAttack | AA robustness (%): 30.42 -> 31.91 | Helped |
| add class-wise global information (GI) to DKL with BA (ablation) | DKL for ImageNet knowledge distillation · ImageNet, ResNet34 teacher to ResNet18 student | Top-1 accuracy (%): 71.80 -> 71.91 | Helped |
| use DKL instead of KL (ablation) | CIFAR-100 adversarial training · CIFAR-100, AutoAttack | AA robustness (%): 30.29 -> 30.20 | No clear effect |
| set alpha/4 to 3, 4, 5, or 6 (ablation) | IKL adversarial training · CIFAR-100 | AA robustness (%): 31.29; 31.33; 31.91; 31.64 | Mixed |
| set beta to 2, 3, 4, or 5 (ablation) | IKL adversarial training · CIFAR-100 | AA robustness (%): 30.95; 31.33; 31.57; 31.91 | Helped |
| set global-information temperature tau to 1, 2, 3, or 4 (ablation) | IKL adversarial training · CIFAR-100 | Clean accuracy (%): 59.99; 63.77; 65.28; 65.76 | Helped |
| replace KL with IKL | TRADES adversarial training · CIFAR-100, AutoAttack at epsilon 8/255 | AA robustness (%): 30.29 -> 31.91 | Helped |
| replace KL with IKL | Knowledge distillation on ImageNet-LT · ImageNet-LT, ResNeXt-101 teacher to ResNet-50 | All-class top-1 accuracy (%): 48.31 -> 49.29 | Helped |
Surprising findings
- Changing the global-information temperature from 1 to 4 raises clean accuracy by about 5.75 points while keeping robustness comparable.
- Cross-entropy and wMSE gradients are asymmetric; when the teacher is detached in KD, the wMSE term has no effect unless gradients are enabled on the student logits.
Limitations
- Experiments limited mainly to image classification on CIFAR and ImageNet; other tasks (OOD robustness, incremental learning) left for future work.
- Hyperparameters alpha, beta and tau were tuned on the evaluation setting (CIFAR-100 AA), with gains of a few points over baselines.
- Robustness gains over previous SOTA are modest (e.g., +0.68% with AutoAugment).
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet ImageNet-LT ADE20K
Benchmarks: CIFAR-10 CIFAR-100 ImageNet RobustBench ImageNet-LT ADE20K
Metrics: top-1 clean accuracy AutoAttack robust accuracy top-1 accuracy training speed (sec/iter)
Baselines: TRADES AWP LBGAT LAS-AT ACAT DM-AT DKD KD ReviewKD OFD CRD FitNet
Builds on: TRADES AWP DKD DM-AT Knowledge Distillation
Closest papers by meaning
- Statistical and Geometrical properties of the Kernel Kullback-Leibler divergence (NeurIPS 2024 · similarity 0.88)
- Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss (ICML 2025 · similarity 0.87)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.87)
- Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation (NeurIPS 2024 · similarity 0.86)
- Better Estimation of the Kullback--Leibler Divergence Between Language Models (NeurIPS 2025 · similarity 0.85)
- DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.