Research map / Architectures and efficiency / RNNs and normalization
Average gradient outer product as a mechanism for deep neural collapse
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that deep neural collapse arises in Deep Recursive Feature Machines via iterated AGOP projection, gives asymptotic and kernel-learning explanations, and argues the AGOP-aligned right singular structure of trained DNN weights drives within-class variability collapse.
Problem
Existing explanations of deep neural collapse rely on data-agnostic unconstrained features models, ignoring how training data and feature learning drive collapse across layers.
Approach
Attributes deep neural collapse to projection onto the average gradient outer product (AGOP) at each layer: Deep RFM, a backprop-free kernel-based method that alternates kernel regression, AGOP computation and random feature maps, exhibits DNC; the right singular structure of trained DNN weights (linked to AGOP via the Neural Feature Ansatz) produces most within-class variability reduction.
Key results
- Deep RFM exhibits exponential convergence to DNC (NC1 and NC2) across layers on CIFAR-10, MNIST and SVHN, with the NC1 improvement attributed entirely to the AGOP projection rather than the random feature map.
- For MLP, ResNet trained with SGD and small initialization, the right singular structure S V^T reduces within-class variability most of the way; NFA correlations at end of training were 0.74±0.17 (ResNet, CIFAR-10) and 0.74±0.13 (MLP, MNIST).
- Parametrized kernel ridge regression optimum over kernel matrices is exactly the neural collapse kernel Y^T Y, giving a non-asymptotic implicit-bias explanation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| project each layer's features using the AGOP square root | Deep RFM · CIFAR-10, MNIST, and SVHN | NC1 | Helped |
| apply the random feature map without the AGOP projection (ablation) | Deep RFM · CIFAR-10, MNIST, and SVHN | NC1 | Hurt |
| use cosine rather than ReLU random features (ablation) | Deep RFM · MNIST, SVHN, and CIFAR-10 | NC1 | Mixed |
| use ReLU rather than cosine random features (ablation) | Deep RFM · MNIST, SVHN, and CIFAR-10 | NC1 | Mixed |
| apply the right singular structure of the weight matrix | MLP and ResNet trained with small initialization · MNIST and CIFAR-10 | NC1 | Helped |
| apply the left singular vectors and nonlinearity after the right singular structure (ablation) | MLP and ResNet trained with small initialization · MNIST and CIFAR-10 | NC1 | No clear effect |
Surprising findings
- Random feature maps (ReLU or cosine) slightly worsen or leave NC1 unchanged; the DNC1 improvement comes entirely from the AGOP projection.
- Unlike DNC1, strong DNC2 progression appears only in the final layer for trained DNNs with small initialization.
Limitations
- Theoretical DNC result for Deep RFM is asymptotic in high dimensions with proportional regime and full-rank assumptions.
- Mechanism for general DNNs is supported by correlation evidence (NFA) rather than a proof; DNC2 does not progress strongly in intermediate layers.
- Experiments limited to small-scale vision datasets (MNIST, SVHN, CIFAR-10) and MLP/VGG/ResNet architectures.
Experimental setup
Datasets: MNIST SVHN CIFAR-10
Benchmarks: MNIST SVHN CIFAR-10
Metrics: NC1 (tr(Sigma_W)/tr(Sigma_B)) NC2 ETF distance Neural Feature Ansatz correlation training loss
Baselines: unconstrained features model deep unconstrained features model
Builds on: Recursive Feature Machine Deep RFM Neural Feature Ansatz unconstrained features model
Closest papers by meaning
- Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model (NeurIPS 2025 · similarity 0.88)
- Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data (NeurIPS 2025 · similarity 0.88)
- The Persistence of Neural Collapse Despite Low-Rank Bias (NeurIPS 2025 · similarity 0.88)
- Neural collapse vs. low-rank bias: Is deep neural collapse really optimal? (NeurIPS 2024 · similarity 0.87)
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.87)
- The Prevalence of Neural Collapse in Neural Multivariate Regression (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.