Research map / Architectures and efficiency / RNNs and normalization
The Prevalence of Neural Collapse in Neural Multivariate Regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that neural multivariate regression exhibits three neural-collapse-like properties (NRC1-3) and explains them via the unconstrained feature model with L2 regularization, showing no collapse without regularization.
Problem
Neural collapse has been characterized only for classification; it is unknown whether regression models trained with MSE exhibit analogous geometric collapse of last-layer features and weights.
Approach
Defines Neural Regression Collapse (feature collapse to the top-n PCA subspace, feature-weight self-duality, and a covariance-determined Gram structure of W) and explains it with the unconstrained feature model for L2 regression, where collapse requires strictly positive regularization.
Key results
- NRC1-NRC3 emerge across all six datasets (MLP and ResNet architectures) during training with weight decay.
- When weight decay is exactly zero there is no collapse; a small increase above zero makes NRC1-3 emerge.
- Theory: with strictly positive regularization, global minima of the regularized UFM satisfy NRC1-NRC3 with gamma = c = lambda_W * lambda_H; without regularization there are infinitely many non-collapsed global minima.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with positive weight decay | neural regression models · Swimmer, Reacher, Hopper, CARLA 1D, CARLA 2D, UTKFace | NRC1 | Helped |
| train with positive weight decay | neural regression models · Swimmer, Reacher, Hopper, CARLA 1D, CARLA 2D, UTKFace | NRC2 | Helped |
| train with positive weight decay | neural regression models · Swimmer, Reacher, Hopper, CARLA 1D, CARLA 2D, UTKFace | NRC3 | Helped |
| remove weight decay (ablation) | neural regression models · Six regression datasets | NRC1-NRC3 | Hurt |
| remove weight decay (ablation) | MLP on small MuJoCo datasets · MuJoCo | test MSE | Hurt |
| increase weight decay slightly above zero (ablation) | neural regression models · Six regression datasets | NRC1-NRC3 | Helped |
| use L2 penalties on features and final-layer weights instead of weight decay, and remove penultimate-layer ReLU (ablation) | UFM-assumption model · Six regression datasets | NRC1-NRC3 | Hurt |
| increase feature regularization while holding final-layer regularization fixed (ablation) | UFM-assumption model · Additional experiments | feature norm | Hurt |
| increase feature regularization while holding final-layer regularization fixed (ablation) | UFM-assumption model · Additional experiments | final-layer weight norm | Helped |
Surprising findings
- Collapse is discontinuous in the regularization: zero weight decay yields no collapse even though small positive weight decay yields full collapse.
- For univariate regression NRC3 is trivially zero and therefore not informative.
Limitations
- Connection between neural regression collapse and generalization is not addressed.
- NRC3 is not meaningful for the univariate case, and the UFM theory is for the L2 loss with a free-feature idealization.
- Experiments use only six datasets and small MLP/ResNet architectures.
Experimental setup
Datasets: MuJoCo Swimmer expert data MuJoCo Reacher expert data D4RL Hopper CARLA driving dataset UTKFace
Benchmarks: Swimmer Reacher Hopper CARLA UTKFace
Metrics: NRC1 NRC2 NRC3 train/test MSE R^2 explained variance ratio
Builds on: neural collapse unconstrained feature model layered-peeled model
Closest papers by meaning
- Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model (NeurIPS 2025 · similarity 0.91)
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.90)
- Neural Collapse Beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime (ICML 2025 · similarity 0.89)
- Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data (NeurIPS 2025 · similarity 0.88)
- Neural Collapse To Multiple Centers For Imbalanced Data (NeurIPS 2024 · similarity 0.88)
- Model Collapse Demystified: The Case of Regression (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.