Research map / Learning theory and algorithms / Kernels and regression theory
The Persistence of Neural Collapse Despite Low-Rank Bias
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends the Sukenik et al. result that deep neural collapse is suboptimal under MSE to deep linear and ReLU unconstrained feature models with cross-entropy loss, proving low-rank bias and showing DNC remains a locally attractive critical point for small regularization.
Problem
Deep neural collapse (DNC) is empirically common in trained networks, yet recent work shows it is suboptimal under MSE loss in deep unconstrained feature models (UFMs). It is unclear why DNC persists in practice under cross-entropy loss, and how low-rank bias shapes the loss landscape.
Approach
Analyzes deep linear and ReLU UFMs with cross-entropy and L2 regularization, showing that for enough separated layers any DNC structure is beaten by a lower-rank diagonally superior construction; global minima become approximately low-rank with exponentially decaying extra singular values, while DNC stays a critical point with PSD Hessian for small regularization and its solution set grows faster
Key results
- In the deep linear UFM with cross-entropy, DNC is not a global minimum for K>=4, L>=3 or K>=6, L=2 (Theorem 1).
- Global minima have all but at most q_K singular values decaying faster than exponentially in L when lambda=o(1/L) (Theorem 3), with q_K at most 2 as an example.
- The number of DNC-structured solutions grows faster with width than low-rank solutions, with the dimension ratio tending to (K-1)/r > 1 (Theorem 5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the low-rank structure instead of DNC | Deep linear UFM · Deep UFM experiments | loss | Helped |
| Increase network width | Deep linear UFM · Deep UFM experiments | probability of DNC | Helped |
| Decrease regularization | Deep linear UFM · Deep UFM experiments | probability of DNC | Helped |
| Use the low-rank solution instead of the DNC bound | ResNet-20 feature map with a fully connected linear head · MNIST and CIFAR-10 | loss | Helped |
| Use a low-rank solution instead of DNC | ResNet-20 feature map with a fully connected ReLU head · CIFAR-10 | loss | Helped |
| Apply standard regularization with weight decay on every parameter | ResNet-20 feature map with a fully connected head · CIFAR-10 | rank structure | Helped |
Surprising findings
- DNC, though suboptimal, remains a critical point with PSD Hessian for small regularization, unlike the L=1 case where the landscape is a strict saddle function.
Limitations
- Analysis is restricted to deep linear UFMs and a ReLU extension under an assumption; omits batch size, initialization, and optimizer effects.
- Theory on imbalanced classes is limited to (R, rho)-STEP imbalance and large L_0 bounds.
- Persistence of DNC is only heuristically explained via solution counting, not via basins of attraction.
Experimental setup
Datasets: MNIST CIFAR-10
Benchmarks: MNIST CIFAR-10
Metrics: loss singular values probability of DNC
Baselines: deep neural collapse solution
Builds on: Sukenik et al. deep UFM MSE analysis Schatten quasi-norm characterization Thrampoulidis et al. imbalanced UFM Papyan et al. neural collapse
Closest papers by meaning
- Neural collapse vs. low-rank bias: Is deep neural collapse really optimal? (NeurIPS 2024 · similarity 0.95)
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.89)
- Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model (NeurIPS 2025 · similarity 0.89)
- Neural Collapse Beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime (ICML 2025 · similarity 0.89)
- Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers (NeurIPS 2025 · similarity 0.89)
- Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.