Research map / Label-efficient and robust learning / Recognition and long-tail
Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Features Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Characterizes global minimizers of cross-entropy training under a non-negative-feature unconstrained features model in class-imbalanced settings, deriving orthogonal class-mean geometry and classifier alignment with scaled-and-centered class-means.
Problem
Neural Collapse geometry for cross-entropy loss under class-imbalanced data is not explicitly characterized; prior UFM analyses give only block structures without magnitudes, and do not cover classifier/feature geometry (NC2, NC3).
Approach
Extends the unconstrained features model to non-negative (ReLU-like) features and CE loss with class-count-dependent weighting, proving that global minimizers have within-class collapse, orthogonal class-means with closed-form norms depending on n_k, and classifiers aligned to scaled-and-centered class-means.
Key results
- Within-class features collapse (NC1) and class-means become mutually orthogonal under non-negative features, with closed-form norms depending on n_k.
- Classifier weights align with scaled-and-centered class-means, with the balanced case recovering the simplex ETF and self-duality.
- Experiments on MLP, VGG11, and ResNet18 trained on imbalanced CIFAR10, CIFAR100, MNIST, and FashionMNIST show NC metrics converging to small values as training progresses.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| progress training with cross-entropy on imbalanced CIFAR10 | MLP, VGG11, and ResNet18 · CIFAR10 imbalanced subset | NC1 feature-collapse metric | Helped |
Surprising findings
- The norm of optimal features is not always negatively correlated with class sample count, since M_i/sqrt(n_i) is not monotone in n_i.
- Classifier angles between majority classes are larger than between minority classes, the opposite of what one might expect from minority-class volume intuition in some settings.
Limitations
- Analysis only covers feature dimension d at least number of classes K; bottleneck case d < K is unaddressed.
- Theory relies on the unconstrained features model, which abstracts away the network's feature mapping.
Experimental setup
Datasets: CIFAR10 CIFAR100 MNIST FashionMNIST
Benchmarks: CIFAR10 imbalanced subset CIFAR100 imbalanced subset MNIST imbalanced subset FashionMNIST imbalanced subset
Metrics: NC1 feature collapse metric NC2 classifier-feature alignment NC2 Gram matrix distance NC3 prediction matrix distance
Baselines: UFM-SVM concurrent CE block-structure analysis
Builds on: unconstrained features model UFM+ (non-negative features) SELI geometry General Orthogonal Frame
Closest papers by meaning
- Neural Collapse To Multiple Centers For Imbalanced Data (NeurIPS 2024 · similarity 0.90)
- Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Frame (NeurIPS 2024 · similarity 0.88)
- Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective (NeurIPS 2024 · similarity 0.88)
- Generalized Neural Collapse for a Large Number of Classes (ICML 2024 · similarity 0.88)
- Unlocking Better Closed-Set Alignment Based on Neural Collapse for Open-Set Recognition (AAAI 2025 · similarity 0.87)
- Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.