Research map / Label-efficient and robust learning / Knowledge distillation
Distilling Balanced Knowledge from a Biased Teacher
Seonghak Kim
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LTKD instead of standard KD/DKD | ResNet32x4 -> ResNet8x4 · CIFAR-100-LT (gamma=100) | tail accuracy: 15.09 -> 27.21 | Improved |
| add LTKD | ResNet50 -> MobileNetV2 · CIFAR-100-LT (gamma=100) | tail and overall accuracy: +8.59 tail, +3.24 overall | Improved |
| add LTKD | ResNet32x4 -> ResNet8x4 · TinyImageNet-LT (gamma=100) | tail accuracy: 9.09 -> 10.48 | Improved |
| add LTKD | ResNet34 -> ResNet18 · ImageNet-LT | tail accuracy: +1.06, +1.75, +2.40 for gamma 5, 10, 20 | Improved |
| add rebalanced cross-group loss (ablation) | LTKD (cross-group only) · CIFAR-100-LT (gamma=20), ResNet32x4-ResNet8x4 | tail accuracy: 38.30 -> 40.51 | Improved |
| add rebalanced cross-group loss (ablation) | LTKD (cross-group only) · CIFAR-100-LT (gamma=20), ResNet32x4-ShuffleNetV1 | overall accuracy: -0.01 overall, tail 30.01 -> 30.68 | No clear effect |
| add cross-group loss alone (ablation) | LTKD · CIFAR-100-LT (gamma=20), ResNet32x4-ResNet8x4 | tail accuracy: +3.70 | Improved |
| add cross-group loss alone (ablation) | LTKD · CIFAR-100-LT (gamma=20), ResNet32x4-ResNet8x4 | overall accuracy: 57.41 -> 56.55 | Hurt |
| add reweighted within-group loss alone (ablation) | LTKD · CIFAR-100-LT (gamma=20), ResNet32x4-ResNet8x4 | tail accuracy: +5.53 | Improved |
| combine cross- and within-group losses (ablation) | LTKD · CIFAR-100-LT (gamma=20), ResNet32x4-ResNet8x4 | tail accuracy: 42.34 -> 49.70 | Improved |
| combine both losses (ablation) | LTKD · CIFAR-100-LT (gamma=20), VGG13-MobileNetV2 | tail and overall accuracy: +9.28 tail, +4.34 overall | Improved |
| increase number of groups from 3 to 10-20 (ablation) | LTKD · CIFAR-100-LT (gamma=100) | overall accuracy: VGG13-VGG8 47.66 -> 48.69 (n=20); R32x4-R8x4 51.08 -> 50.41 (n=100) | Mixed |
Closest papers by meaning
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.91)
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.89)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.89)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.88)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.88)
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
