Research map / Label-efficient and robust learning / Knowledge distillation
Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes multi-round self-distillation in linear probing as label averaging over Gram-matrix-correlated instances, derives conditions for 100% population accuracy under label noise, and proposes a single-round PLL student trained on the teacher's top-two softmax outputs.
Problem
Whether self-distillation helps linear probing on a fixed feature extractor (where feature learning explanations do not apply), how repeated rounds affect outputs under label noise, and whether multi-round gains can be achieved in a single round at lower cost.
Approach
Under a linearized softmax, multi-round self-distillation is a repeated averaging of given labels through a transformed Gram matrix, so predictions cluster by ground-truth class; a single-round student trained on the teacher's top-two refined partial labels (weight 1/2 each) reproduces this benefit.
Key results
- Multi-round self-distillation with linear probing on six image datasets shows test accuracy increasing with distillation rounds under label noise.
- The PLL student model outperforms multi-round distillation models at corruption rates eta >= 0.6 in a single round, per the paper's experiments.
- Theory gives a sufficient condition for 100% population accuracy that becomes easier to satisfy as the round t grows, and the PLL condition ([C]_{k,k} > [C]_{k,k'}) is weaker than the multi-round condition.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of self-distillation rounds | Linear probing with multi-round self-distillation · Six image-classification benchmarks across label corruption scenarios | test accuracy | Helped |
| Replace multi-round soft-target distillation with one-round top-two partial labels | Self-distillation · Six image-classification benchmarks, η ≥ 0.6 | test accuracy | Helped |
| Increase the candidate set beyond the top two teacher predictions (ablation) | PLL student · CIFAR-100 and other image-classification datasets | test accuracy | Hurt |
Surprising findings
- Cross-entropy with equally weighted two-hot partial labels was unstable, so the authors switched to generalized cross entropy for the PLL student.
Limitations
- Softmax is replaced by a linear approximation, valid only for small logits; approximation error is validated numerically rather than theoretically.
- Feature correlation is assumed to depend only on ground-truth class (block-wise Gram structure), which is only empirically justified.
- Experiments use a fixed feature extractor (linear probing) rather than full network training.
Experimental setup
Datasets: CIFAR-100 Caltech-101 Caltech-256 Flowers-102 Food-101 StanfordCars ImageNet
Benchmarks: CIFAR-100 Caltech-101 Caltech-256 Flowers-102 Food-101 StanfordCars
Metrics: test accuracy accuracy improvement over teacher
Baselines: multi-round self-distillation teacher model
Builds on: knowledge distillation born-again networks partial label learning generalized cross entropy
Closest papers by meaning
- Distilling Reliable Knowledge for Instance-Dependent Partial Label Learning (AAAI 2024 · similarity 0.89)
- The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model (NeurIPS 2025 · similarity 0.88)
- A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.88)
- Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classification (AAAI 2024 · similarity 0.87)
- Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations (NeurIPS 2024 · similarity 0.86)
- M2SD:Multiple Mixing Self-Distillation for Few-Shot Class-Incremental Learning (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.