Research map / Architectures and efficiency / RNNs and normalization
Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves neural collapse (NC1, NC2, NC3) for end-to-end trained deep networks ending in at least two linear layers, via balancedness plus interpolation and bounded conditioning, with gradient descent plus weight decay as the training setting.
Problem
Neural collapse has mostly been analyzed in the unconstrained features model, which is data-agnostic; it is unclear whether NC emerges in end-to-end training of DNNs.
Approach
Sufficient conditions for NC (low training error, approximate balancedness of consecutive linear layers, bounded conditioning of the features and linear head) are derived, then shown to hold for L2-regularized gradient descent with a wide first layer, at global minimizers, or under large learning rates (edge of stability).
Key results
- Theory: NC1 bound vanishes as epsilon_1, epsilon_2 go to 0 under GD with weight decay for networks with a wide first layer, pyramidal topology and smooth activation.
- Experiments: NC2 improves progressively as the linear head gets deeper, e.g. MLP on MNIST with 1 to 5 linear layers and ResNet20 on CIFAR10 with 1 to 6 linear layers.
- Linear layers become increasingly balanced throughout training, while non-linear layers plateau at larger balancedness values.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the depth of the linear head | MLP and ResNet20 · MNIST and CIFAR10 | NC2 in the last layer | Helped |
| increase the depth of the linear head | MLP and ResNet20 · MNIST and CIFAR10 | NC2 at the input layer to the linear head | Hurt |
| increase the depth of the linear head | MLP and ResNet20 · MNIST and CIFAR10 | NC1 | No clear effect |
| train with weight decay | linear layers · Training | linear-layer balancedness | Helped |
| increase the depth of the nonlinear part | nonlinear head layers · Nonlinear head depth 4–12 | balancedness | Helped |
| increase the depth of the nonlinear part | nonlinear head layers · Nonlinear head depth 4–12 | negativity | Helped |
Surprising findings
- NC2 gets slightly worse in the input layer to the linear head as the head deepens, while improving in the last layer.
- The NC1 metric shows no strong dependence on the number of linear head layers.
Limitations
- Theoretical guarantees rely on a wide first layer, pyramidal topology and smooth activations; ReLU is only approximated.
- Large-learning-rate and global-minimizer results rely on assumptions (e.g., edge of stability, bounded NTK) not fully proven in general.
Experimental setup
Datasets: MNIST CIFAR10
Benchmarks: MNIST CIFAR10
Metrics: NC1 (tr(Sigma_W)/tr(Sigma_B)) NC2 (condition number of class-mean matrix) NC3 (average cosine similarity of features and classifier we balancedness negativity
Builds on: unconstrained features model Polyak-Lojasiewicz inequality analysis of deep linear and nonlinear networks edge of stability
Closest papers by meaning
- Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data (NeurIPS 2025 · similarity 0.92)
- Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers (NeurIPS 2025 · similarity 0.90)
- The Prevalence of Neural Collapse in Neural Multivariate Regression (NeurIPS 2024 · similarity 0.90)
- Generalized Neural Collapse for a Large Number of Classes (ICML 2024 · similarity 0.90)
- Neural Collapse Beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime (ICML 2025 · similarity 0.90)
- The Persistence of Neural Collapse Despite Low-Rank Bias (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.