Research map / Label-efficient and robust learning / Knowledge distillation
Enhancing Class-Imbalanced Learning with Pre-Trained Guidance through Class-Conditional Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CCKD and ACCKD, which distill a CLIP-based teacher's class-conditional distribution p(x|y) into a student on imbalanced data, using mixup/CutMix balanced synthetic data and feature imitation.
Problem
Class-imbalanced learning suffers from poor generalization of minority-class features because minority samples poorly represent p(x|y); standard knowledge distillation transfers p(y|x) and fails to transfer useful class-conditional knowledge from pre-trained teachers on imbalanced data.
Approach
Reformulate distillation so the student learns the teacher's class-conditional distribution p(x|y) by correcting label priors (Bayes decomposition), then augment with a class-balanced mixup/CutMix dataset and a cosine feature-imitation loss.
Key results
- On ImageNet-LT with ResNet-18 student, ACCKD reaches 55.4% top-1 vs 44.6% for the LA-only student (+8.8%).
- On iNaturalist 2018, ACCKD with the AF+LA teacher reaches 66.7% vs 57.6% for student LA; CCKD and ACCKD improve over comparators by 2.6% and 5.8% average with NCM/AF+LA teachers.
- Averaged over datasets, the paper reports a 7.4% accuracy improvement over the baseline (abstract).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ACCKD with a zero-shot teacher instead of Student (LA) | ResNet-32 · CIFAR-100-LT | top-1 accuracy (All): 45.5 -> 50.1 | Helped |
| Use ACCKD with a zero-shot teacher instead of Student (LA) | ResNet-18 · ImageNet-LT | top-1 accuracy (All): 44.6 -> 54.5 | Helped |
| Use ACCKD with an AF+LA teacher instead of Student (LA) | ResNet-18 · iNaturalist 2018 | top-1 accuracy (All): 57.6 -> 66.7 | Helped |
| Use ACCKD rather than KD with a zero-shot teacher | ResNet-18 · ImageNet-LT | top-1 accuracy (All): 49.8 -> 54.5 | Helped |
| Use ACCKD rather than KD with an AF+LA teacher | ResNet-18 · iNaturalist 2018 | top-1 accuracy (All): 61.1 -> 66.7 | Helped |
| Add feature imitation on training data (ablation) | ACCKD · ImageNet-LT | top-1 accuracy: 52.8 -> 53.4 | Helped |
| Add supervised loss on synthesis data (ablation) | ACCKD · ImageNet-LT | top-1 accuracy: 53.4 -> 54.4 | Helped |
| Add CCKD loss on synthesis data (ablation) | ACCKD · ImageNet-LT | top-1 accuracy: 54.4 -> 54.7 | Helped |
| Add feature imitation on synthesis data (ablation) | ACCKD · ImageNet-LT | top-1 accuracy: 54.7 -> 55.0 | Helped |
| Complete the full ACCKD loss configuration (ablation) | ACCKD · ImageNet-LT | top-1 accuracy: 55.0 -> 55.4 | Helped |
Surprising findings
- Standard KD and class-weighted KD perform worse than the student trained with logit adjustment alone on several settings, and the zero-shot teacher hurts on iNaturalist 2018 for most baselines.
Limitations
- Student models cannot be fairly compared to large teachers, so the teacher serves only as a reference upper bound.
- Zero-shot teacher performs poorly on iNaturalist 2018, limiting the approach when the pre-trained model is weakly aligned with the target domain.
- Relies on estimates of p_t(y) from a validation set that matches test distribution; falls back to resampling otherwise.
Experimental setup
Datasets: CIFAR-100-LT ImageNet-LT iNaturalist 2018
Benchmarks: CIFAR-100-LT ImageNet-LT iNaturalist 2018
Metrics: top-1 accuracy many-shot accuracy medium-shot accuracy few-shot accuracy
Baselines: Knowledge Distillation (KD) Weighted KD DiVE Student CE Student LA Zero-shot CLIP teacher NCM teacher AdaptFormer+LA teacher
Builds on: knowledge distillation logit adjustment DiVE AdaptFormer Mixup CutMix
Closest papers by meaning
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.88)
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.87)
- Maintaining Fairness in Logit-based Knowledge Distillation for Class-Incremental Learning (AAAI 2025 · similarity 0.87)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.87)
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.87)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.