Research map / Optimization / Implicit bias and generalization
Neural Collapse Beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows, for a three-layer mean-field network trained by gradient flow, that near-stationary low-loss points satisfy NC1 and that well-separated data yields both NC1 and vanishing test error.
Problem
Neural collapse (NC1) has been studied mainly under the data-agnostic unconstrained features model, which ignores the data distribution and training dynamics, so its link to test error and gradient training is unclear.
Approach
Analyze NC1 in a three-layer network (infinite-width mean-field two-layer feature map followed by a linear layer) with entropic and L2 regularization: any epsilon-stationary point with small empirical loss has NC1 with a bound controlled by loss and gradient norm; gradient flow reaches such points, and for well-separated data test error also vanishes.
Key results
- Theorem gives NC1 bound in terms of residual loss and gradient norm: NC1 vanishes as the empirical loss and stationarity error shrink for sufficiently small regularization.
- Gradient flow on the regularized free energy reaches O(beta^{-1}) free energy and NC1 <= delta_0 for long training times (Corollary).
- For (tau,M)-linearly separable data, test error is bounded by C log(n)/sqrt(n) plus a generalization term while NC1 holds, showing NC1 and vanishing test error co-occur.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train the three-layer mean-field network with gradient flow to small loss and gradient norm | three-layer mean-field network · balanced classification | NC1 | Helped |
| use two-stage gradient flow on linearly separable data | three-layer mean-field network · bounded, (τ,M)-linearly separable data | test error: C_{10} \log (C_{11} n/\delta_0) \sqrt{\frac{1}{2n}} + 6q \sqrt{\frac{\log(2/\delta)}{n}} | Helped |
| use two-stage gradient flow on linearly separable data | three-layer mean-field network · bounded, (τ,M)-linearly separable data | NC1: δ_0 | Helped |
| train without achieving balancedness | three-layer network · MNIST (3 relabeled classes) | NC1 | Helped |
| train without achieving balancedness | three-layer network · CIFAR-100 (3 super-classes) | NC1 | Helped |
Surprising findings
- Stationary points of the regularized free energy are not balanced, so prior balancedness-based NC1 analyses do not apply; experiments on MNIST and CIFAR-100 show NC1 emerges despite persistent imbalance (normalized balancedness 0.28-0.85 at convergence).
Limitations
- Analysis restricted to a three-layer model with linear final layer and no nonlinearity between last two layers.
- Test error result requires well-separated (linearly separable) data distributions and truncation of the second layer.
- Requires small regularization and is established for gradient flow over long training times rather than practical finite-step training.
Experimental setup
Datasets: MNIST CIFAR-100
Benchmarks: MNIST CIFAR-100
Metrics: NC1 metric empirical training loss gradient norm normalized balancedness test error
Builds on: unconstrained features model mean-field regime analysis of two-layer networks Wasserstein gradient flow
Closest papers by meaning
- Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data (NeurIPS 2025 · similarity 0.90)
- Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model (NeurIPS 2025 · similarity 0.90)
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.90)
- The Persistence of Neural Collapse Despite Low-Rank Bias (NeurIPS 2025 · similarity 0.89)
- The Prevalence of Neural Collapse in Neural Multivariate Regression (NeurIPS 2024 · similarity 0.89)
- Neural Collapse To Multiple Centers For Imbalanced Data (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.