Research map / Label-efficient and robust learning / Recognition and long-tail
For Better or For Worse? Learning Minimum Variance Features With Label Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically and empirically that label smoothing and Mixup (label augmentation) bias linear and general models toward minimum-variance features, which correlates with better CIFAR performance but makes them vulnerable to low-variance spurious correlations.
Problem
Understanding why label smoothing and Mixup improve classification performance, and what feature-learning bias label augmentation induces relative to standard ERM with weight decay.
Approach
Label augmentation losses (label smoothing, Mixup) provably make linear classifiers learn only the low-variance feature dimensions, whereas weight decay with cross-entropy keeps weight on the more separable high-variance dimensions; general multi-class lower bounds tie these losses to model output variance.
Key results
- Linear models trained with label smoothing or Mixup have weight concentrated on low-variance dimensions; theorems show ||w_H|| < O(||Sigma_YX,L||) for minimizers, while weight decay gives ||w_H||^2 >= 1/2 ||w||^2.
- On ResNet-18 CIFAR-10/100, adding label smoothing or Mixup to ERM+weight decay reduces test error and significantly lowers penultimate-layer activation variance.
- On spuriously correlated binary CIFAR and colored MNIST, label smoothing and Mixup models have high test error across all hyperparameters while weight decay achieves significantly lower test error for all lambda > 0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add label smoothing to ERM + weight decay | ResNet-18 · CIFAR-10 and CIFAR-100 test | test error | Helped |
| add Mixup to ERM + weight decay | ResNet-18 · CIFAR-10 and CIFAR-100 test | test error | Helped |
| add label smoothing | ResNet-18 · CIFAR-10 and CIFAR-100 test | penultimate-layer activation variance | Helped |
| add Mixup | ResNet-18 · CIFAR-10 and CIFAR-100 test | penultimate-layer activation variance | Helped |
| use label smoothing instead of weight decay | Logistic regression · Binary CIFAR-10 and CIFAR-100 with spurious training dimension | test error | Hurt |
| use Mixup instead of weight decay | Logistic regression · Binary CIFAR-10 and CIFAR-100 with spurious training dimension | test error | Hurt |
| use label smoothing instead of weight decay | 2-layer ReLU network · Colored MNIST test with permuted background colors | test error | Hurt |
| use Mixup instead of weight decay | 2-layer ReLU network · Colored MNIST test with permuted background colors | test error | Hurt |
Surprising findings
- Label smoothing and Mixup can fail badly under low-variance spurious correlations, performing worse than weight decay even though they generalize better on standard benchmarks.
- Even a small amount of weight decay added to label smoothing or Mixup does not change the spurious-feature failure.
Limitations
- Theory for feature learning is restricted to linear models with compact support and binary classification; general multi-class results only give lower bounds on output variance.
- Empirical link between low variance features and generalization is correlational, not causal.
- Experiments use Adam/AdamW with a fixed learning rate and limited training, so they may not reach max-margin convergence.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 MNIST (colored variant) synthetic 2-D and high-dimensional data
Benchmarks: CIFAR-10 CIFAR-100 colored MNIST binary CIFAR-10/100 reductions
Metrics: test error penultimate layer activation variance output probability variance norm ratio of first vs remaining weights
Baselines: ERM with weight decay cross-entropy label smoothing Mixup
Builds on: Mixup label smoothing neural collapse
Closest papers by meaning
- Does label smoothing mitigate label noise? (ICML 2020 · similarity 0.88)
- Augmentation Strategies for Learning With Noisy Labels (CVPR 2021 · similarity 0.88)
- From Label Smoothing to Label Relaxation (AAAI 2021 · similarity 0.87)
- Does Label Smoothing Help Deep Partial Label Learning? (ICML 2024 · similarity 0.86)
- MaxSup: Overcoming Representation Collapse in Label Smoothing (NeurIPS 2025 · similarity 0.85)
- Training Deep Neural Networks with Virtual Smoothing Classes (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.