Research map / Label-efficient and robust learning / Recognition and long-tail
MaxSup: Overcoming Representation Collapse in Label Smoothing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decomposes the label smoothing loss into a regularization term and an error-amplification term, and proposes MaxSup, which penalizes the top-1 logit instead of the ground-truth logit to improve accuracy and feature diversity.
Problem
Label smoothing induces overconfidence on misclassified samples and compresses intra-class feature variation, but the mechanism behind these failures was unclear.
Approach
Rewrites the LS loss at the logit level, showing an error-amplification term that penalizes the gap to larger non-ground-truth logits on misclassified samples; MaxSup replaces the ground-truth penalty with a penalty on the top-1 logit, giving uniform regularization for correct and incorrect predictions.
Key results
- On ImageNet-1K with DeiT-Small, MaxSup reaches 76.49% top-1 vs 76.08% for label smoothing (mean over runs).
- On ImageNet-1K with ResNet-50, MaxSup achieves 77.69% vs 76.91% for label smoothing.
- MaxSup retains linear-probe CIFAR-10 transfer accuracy of 0.810, close to baseline 0.814, while label smoothing drops to 0.746.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Label Smoothing's ground-truth-logit penalty with MaxSup's top-1-logit penalty (ablation) | DeiT-Small · ImageNet-1K | accuracy: 75.91 -> 76.12 | Helped |
| Remove error amplification while retaining regularization (ablation) | Label Smoothing on DeiT-Small · ImageNet-1K | accuracy: 75.91 -> 75.98 | Helped |
| Add the error-amplification term normalized over larger logits (ablation) | DeiT-Small · ImageNet-1K | accuracy: 74.21 -> 73.63 | Hurt |
| Add the top-logit error-amplification term (ablation) | DeiT-Small · ImageNet-1K | accuracy: 74.21 -> 73.69 | Hurt |
| Increase alpha from 0.1 to an increasing schedule (ablation) | MaxSup on DeiT-Small · ImageNet-1K | accuracy: 76.12 -> 76.58 | Helped |
| Increase alpha from 0.1 to an increasing schedule (ablation) | Label Smoothing on DeiT-Small · ImageNet-1K | accuracy: 75.91 -> 76.16 | Helped |
| Use MaxSup instead of OLS | ResNet-50 · ImageNet-1K | top-1 accuracy: 77.23±0.21 -> 77.69±0.07 | Helped |
| Use MaxSup instead of Label Smoothing | ResNet-18 · CIFAR-100 | top-1 accuracy: 77.05±0.17 -> 77.82±0.15 | Helped |
| Pretrain with MaxSup instead of Label Smoothing | DeiT-Small with UperNet · ADE20K | mIoU: 42.4 -> 42.8 | Helped |
| Pretrain with MaxSup instead of the baseline | linear probe · CIFAR-10 | accuracy: 0.814 -> 0.810 | Hurt |
| Use MaxSup instead of Label Smoothing | CIFAR-10-LT ResNet-32 · test, imbalance ratio 100 | overall accuracy: 76.4 -> 76.4 | No clear effect |
| Use MaxSup instead of Label Smoothing | CIFAR-10-LT ResNet-32 · test, imbalance ratio 50 | medium-shot accuracy: 75.4 -> 73.4 | Hurt |
Surprising findings
- Label smoothing's ImageNet gains come entirely from the regularization term; the error-amplification term reduces accuracy.
- Logit Penalty improves ImageNet accuracy in some settings but degrades transfer accuracy.
Limitations
- Potential effects on knowledge distillation and training dynamics remain unexplored.
- Interaction with l2 regularization and low-rank feature bias is an open question.
- Long-tailed classification does not fully resolve minority-class performance (low-shot accuracy is 0.0).
Experimental setup
Datasets: ImageNet-1K CIFAR-100 CUB-200-2011 Stanford Cars ADE20K CIFAR-10-LT CIFAR-10-C CIFAR-10 Flowers Foods Pets
Benchmarks: ImageNet-1K CIFAR-100 CUB-200-2011 Stanford Cars ADE20K CIFAR-10-LT CIFAR-10-C
Metrics: top-1 accuracy mIoU NLL ECE intra-class variation R^2 inter-class separability linear-probe accuracy
Baselines: Label Smoothing Online Label Smoothing Zipf Label Smoothing Logit Penalty Confidence Penalty Adaptive Label Smoothing Logit Normalization Focal Loss
Builds on: Label Smoothing Online Label Smoothing Zipf Label Smoothing Logit Penalty
Closest papers by meaning
- Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It (ICLR 2025 · similarity 0.90)
- Training Deep Neural Networks with Virtual Smoothing Classes (AAAI 2025 · similarity 0.90)
- From Label Smoothing to Label Relaxation (AAAI 2021 · similarity 0.89)
- Does label smoothing mitigate label noise? (ICML 2020 · similarity 0.88)
- The Devil Is in the Margin: Margin-Based Label Smoothing for Network Calibration (CVPR 2022 · similarity 0.88)
- MLC-NC: Long-Tailed Multi-Label Image Classification Through the Lens of Neural Collapse (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.