Research map / Architectures and efficiency / RNNs and normalization
Neural collapse vs. low-rank bias: Is deep neural collapse really optimal?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that deep neural collapse (DNC2, orthogonal class means) is not optimal for multi-class, deep, non-linear deep unconstrained features models, due to a low-rank bias of multi-layer weight decay, and constructs a strongly regular graph solution that beats DNC while within-class collapse (DNC1) remains optimal.
Problem
Whether deep neural collapse (DNC) is the optimal solution of the deep unconstrained features model (DUFM) for multi-class, multi-layer, non-linear networks, where prior theory covered only linear, two-layer, or binary cases.
Approach
A low-rank bias from multi-layer L2 regularization (representation cost equals a Schatten-2/L quasi-norm) makes low-rank solutions built on strongly regular (triangular) graphs achieve lower loss than DNC for L>=4, K>=6 or L=3, K>=10; within-class collapse is still provably optimal.
Key results
- For L>=4 and K>=6 (or L=3, K>=10), the SRG construction has strictly lower loss than DNC, with loss ratio O(K^((3-L)/(2(L+1)))).
- In 4-DUFM with K=10, learned solutions have intermediate ranks between 5 and 8 versus rank K=10 for DNC.
- In end-to-end ResNet20 training on CIFAR-10 with standard regularization, rank in the third layer ranges between 8 and 9, while DNC rank is K=10; all layers still show DNC1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the low-rank SRG construction instead of DNC | L-DUFM · K≥6 and L≥4 or K≥10 and L=3 | loss: L_SRG < L_DNC | Helped |
| Use the low-rank SRG construction instead of DNC | L-DUFM · K→∞, with L≥4 | SRG-to-DNC loss ratio: O(K^{\frac{3-L}{2(L+1)}}) | Helped |
| Train with gradient descent rather than use the DNC solution | 4-DUFM, K=10 · DUFM training | loss | Helped |
| Increase the number of classes | L-DUFM · DUFM training | prevalence of low-rank solutions | Helped |
| Increase the number of layers | L-DUFM · DUFM training | prevalence of low-rank solutions | Helped |
| Increase weight decay | Gradient descent on L-DUFM · DUFM training | rank at convergence | Hurt |
| Increase width | Gradient descent on L-DUFM · DUFM training | rank at convergence | Hurt |
| Increase width | Gradient descent on L-DUFM · DUFM training | loss | Hurt |
| Use DUFM-like regularization on a ResNet20 with an MLP head | ResNet20 with 4-layer MLP head · CIFAR10 | loss | Helped |
| Use standard weight regularization rather than DUFM-like regularization | ResNet20 with MLP head · CIFAR-10 and MNIST | rank of class-mean matrices: 8 and 9; 5 and 7; K=10 | Mixed |
| Use a small weight decay | Gradient descent on L-DUFM · DUFM training | finding a DNC solution | Helped |
| Use a small width | Gradient descent on L-DUFM · DUFM training | finding a low-rank solution | Helped |
Surprising findings
- Deep neural collapse (orthogonal class-mean frame, DNC2) is not optimal for the DUFM once there are multiple classes and at least three or four layers, contrary to prior theory.
- A larger network width leads gradient descent toward DNC solutions that have higher loss than low-rank solutions.
Limitations
- Analysis is limited to the MSE loss; cross-entropy extension is conjectured, not proven.
- Optimality of DNC1 across all layers is proven only for the last layer or approximately with relaxed ReLU; DNC2 suboptimality in the last layer is supported empirically only.
- Theoretical proof requires K>=6 with L>=4 or K>=10 with L=3, so small-class cases are only covered experimentally.
Experimental setup
Datasets: MNIST CIFAR-10
Benchmarks: MNIST CIFAR-10
Metrics: DNC1 metric tr(Sigma_W)/tr(Sigma_B) condition number of class-mean matrices loss ratio SRG/DNC effective rank of class-mean matrices
Baselines: deep neural collapse solution unconstrained features model
Builds on: unconstrained features model deep unconstrained features model Schatten-p quasi-norm variational form
Closest papers by meaning
- The Persistence of Neural Collapse Despite Low-Rank Bias (NeurIPS 2025 · similarity 0.95)
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.89)
- Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model (NeurIPS 2025 · similarity 0.89)
- Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data (NeurIPS 2025 · similarity 0.89)
- Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers (NeurIPS 2025 · similarity 0.88)
- Average gradient outer product as a mechanism for deep neural collapse (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.