Research map / Label-efficient and robust learning / Recognition and long-tail
Generalized Neural Collapse for a Large Number of Classes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends neural collapse to large numbers of classes (K > d+1) by introducing generalized neural collapse via softmax codes, with empirical checks on ResNet/DenseNet/ResNeXt and theory under a spherically constrained unconstrained feature model.
Problem
Neural collapse has been characterized mainly when the number of classes is small relative to feature dimension; many practical settings (face recognition, language models, retrieval) have far more classes than feature dimensions.
Approach
Replaces the simplex ETF with a softmax code (points on the sphere maximizing the minimum one-vs-rest distance) as the target classifier geometry, defines generalized neural collapse (GNC1-3), and proves that under a HardMax limit of CE with spherical constraints and vanishing temperature, the optimal classifier is a softmax code, with within-class collapse and self-duality holding under condition
Key results
- Fixing the classifier to class-mean features (CMF) on fine-tuning CIFAR10 gives ID accuracy 98.00% vs 97.00% and OOD (STL10) 90.67% vs 87.42% for standard fine-tuning.
- With d=2 and four CIFAR10 classes, a fixed assignment placing Cat and Dog far apart gives 89.95% test accuracy vs 91.90% and 92.13% for other assignments.
- The softmax code one-vs-rest distance is theoretically shown to be bounded above and below by expressions in K and d, implying feature dimension needs roughly log(K^2/rho) for margin rho.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the CMF classifier instead of fine-tuning a randomly initialized classifier | Pretrained ResNet50 · CIFAR10 fine-tuning (ID) | accuracy: 97.00% -> 98.00% | Helped |
| Use the CMF classifier instead of fine-tuning a randomly initialized classifier | Pretrained ResNet50 · STL10 fine-tuning (OOD) | accuracy: 87.42% -> 90.67% | Helped |
| Assign semantically similar class pairs closer together rather than fixing them far apart | ResNet18 classifier · CIFAR10, 4 classes | test accuracy: 89.95% vs 91.90% and 92.13% | Helped |
| Assign semantically similar class pairs closer together rather than fixing them far apart | ResNet18 classifier · CIFAR10, 6 classes | test accuracy: 86.37% vs 91.60% and 91.95% | Helped |
| Use a smaller temperature | Spherical-constraint networks · BUPT-CBFace-50 | GNC2 | Helped |
| Use a relatively small temperature | ResNet18 · CIFAR100 | nearest-centroid classifier accuracy: 100% | Helped |
Surprising findings
- Standard training with a learnable classifier consistently converges to the same class assignment (Cat-Dog close), and forcing dissimilar classes apart hurts test accuracy, suggesting an implicit regularization of class semantic similarity.
Limitations
- Theory uses the HardMax limit and unconstrained features; the equivalence of Tammes and softmax code is proven only for d=2 or K<=d+1.
- Conjecture that Tammes and softmax code are equivalent in general remains open.
Experimental setup
Datasets: CIFAR100 CIFAR10 Tiny-ImageNet BUPT-CBFace-50 ImageNet STL10 MoCo v2 pretraining
Benchmarks: CIFAR100 CIFAR10 STL10 ImageNet Tiny-ImageNet
Metrics: GNC1 within-class variability GNC2 one-vs-rest margin GNC3 self-duality test accuracy out-of-distribution accuracy
Baselines: learnable linear classifier fine-tuning with linear probe
Builds on: neural collapse unconstrained feature model layer-peeled model
Closest papers by meaning
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.90)
- Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data (NeurIPS 2025 · similarity 0.89)
- Neural Collapse To Multiple Centers For Imbalanced Data (NeurIPS 2024 · similarity 0.89)
- Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model (NeurIPS 2025 · similarity 0.89)
- Neural Collapse Beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime (ICML 2025 · similarity 0.88)
- Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Frame (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.