Research map / Label-efficient and robust learning / Recognition and long-tail
Pushing Boundaries: Mixup's Influence on Neural Collapse
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically shows mixup induces a distinctive last-layer geometry (same-class activations on simplex ETF, different-class activations forming calibration-linked channels) and derives optimal features under an adapted unconstrained features model.
Problem
Understanding what geometric configuration of last-layer activations mixup induces, distinct from Neural Collapse, and whether it explains mixup's gains in calibration and generalization.
Approach
Mixup's last-layer features do not follow simple convex combinations of class means; same-class mixed features align with the classifier as a simplex ETF while different-class mixed features form channels along decision boundaries whose density depends on lambda. The authors adapt the unconstrained features model to the mixup loss to derive optimal features and link the geometry to reduced confide
Key results
- Mixup improves CIFAR10 test accuracy for WideResNet-40-10 (96.2 to 97.30) and ViT-B/4 (86.92 to 92.56), with ViT CIFAR100 rising from 59.95 to 69.83.
- CIFAR10 ECE for ViT-B/4 drops from 0.122 (baseline) to 0.014 with mixup (alpha=1.0), while WideResNet ECE rises from 0.024 to 0.077 at alpha=1.0 and falls to 0.013 at alpha=0.4.
- Theory: optimal same-class mixup features are independent of lambda and align with the classifier as a simplex ETF; different-class features are lambda-dependent linear combinations of classifier rows.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add mixup (α=1) | WideResNet-40-10 · FashionMNIST test | accuracy: 95.10 -> 94.21 | Hurt |
| add mixup (α=1) | WideResNet-40-10 · CIFAR10 test | accuracy: 96.2 -> 97.30 | Helped |
| add mixup (α=1) | WideResNet-40-10 · CIFAR100 test | accuracy: 80.03 -> 81.42 | Helped |
| add mixup (α=1) | ViT-B/4 · FashionMNIST test | accuracy: 93.71 -> 94.24 | Helped |
| add mixup (α=1) | ViT-B/4 · CIFAR10 test | accuracy: 86.92 -> 92.56 | Helped |
| add mixup (α=1) | ViT-B/4 · CIFAR100 test | accuracy: 59.95 -> 69.83 | Helped |
| add mixup (α=1.0) | WideResNet-40-10 · CIFAR10 | ECE: 0.024 -> 0.077 | Hurt |
| add mixup (α=0.4) | WideResNet-40-10 · CIFAR10 | ECE: 0.024 -> 0.013 | Helped |
| add mixup (α=1.0) | ViT-B/4 · CIFAR10 | ECE: 0.122 -> 0.014 | Helped |
| add mixup (α=0.4) | ViT-B/4 · CIFAR10 | ECE: 0.122 -> 0.019 | Helped |
| fix the classifier as a simplex ETF during mixup training (ablation) | mixup-trained network · CIFAR10 test | accuracy | No clear effect |
Surprising findings
- Mixup with MSE loss yields features that are simple convex combinations of class features, whereas cross-entropy mixup yields a markedly more complex configuration.
- Mixup ECE for WideResNet-40-10 with alpha=1.0 (0.077) is worse than the baseline (0.024), contrary to the general calibration benefit of mixup.
Limitations
- Theoretical features differ from empirical ones (hexagonal vs elongated channels), requiring an ad hoc amplification to match
- Analysis restricted to three-class subsets and a few architecture/dataset pairs
- Calibration explanation is qualitative and relies on the geometric picture
Experimental setup
Datasets: FashionMNIST CIFAR10 CIFAR100
Benchmarks: FashionMNIST CIFAR10 CIFAR100
Metrics: test accuracy expected calibration error
Baselines: baseline ERM (no mixup) MSE mixup
Builds on: mixup neural collapse unconstrained features model manifold mixup
Closest papers by meaning
- mixup: Beyond Empirical Risk Minimization (ICLR 2018 · similarity 0.89)
- On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks (NeurIPS 2019 · similarity 0.88)
- The Benefits of Mixup for Feature Learning (ICML 2023 · similarity 0.88)
- Catch-Up Mix: Catch-Up Class for Struggling Filters in CNN (AAAI 2024 · similarity 0.86)
- Selective Mixup Helps with Distribution Shifts, But Not (Only) because of Mixup (ICML 2024 · similarity 0.85)
- Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Frame (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.