Research map / Architectures and efficiency / RNNs and normalization
Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that globally optimal deep regularized ResNets and transformers with LayerNorm (one linear layer per MLP block, or two with vanishing regularization) are approximately neurally collapsed, by reducing end-to-end training to a generalized unconstrained features model.
Problem
Neural collapse has been theoretically shown mainly for data-agnostic unconstrained features models or shallow/MLP networks; it is unclear whether end-to-end global optima of modern deep architectures (ResNets with LayerNorm, transformers) exhibit collapse.
Approach
Reduce end-to-end training of sufficiently deep regularized ResNets/transformers to a data-aware generalized unconstrained features model by explicitly constructing deep networks whose regularization cost vanishes as depth grows, so global optima approach the GUFM optima, which are collapsed for CE and MSE losses.
Key results
- For L-RN1 and L-Tx1 with constant regularization, all global optima become approximately collapsed as depth L grows, with distance to perfect collapse decreasing with depth (rate roughly O~(L^{-1/2})).
- For L-RN2 and L-Tx2 with depth-vanishing regularization of the MLP blocks (lambda(L)=o(1/log L)), the same approximate collapse result holds.
- Experiments on MNIST, CIFAR10 and IMDB with depths 2 to 34 show NC1, NC2 and NC3 metrics improve with depth, roughly log-linearly for ResNets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase depth | ResNet and transformer models · CIFAR10, IMDB, and MNIST | NC1 | Helped |
| increase depth | ResNet and transformer models · CIFAR10, IMDB, and MNIST | NC2 | Helped |
| increase depth | ResNet and transformer models · CIFAR10, IMDB, and MNIST | NC3 | Helped |
Surprising findings
- Constant regularization with two linear layers per block does not yield collapse in the analysis; the authors give a concrete dataset where collapse cannot be reached under vanishing regularization of both kinds.
Limitations
- Two-linear-layer blocks with constant regularization remain open; collapse is not asymptotically reached for all datasets.
- Transformer results require labels uniquely determined by context, so language pretraining with multiple continuations is excluded.
- Theory covers only last-layer geometry and asymptotic depth; the rate of convergence is not tight.
Experimental setup
Datasets: MNIST CIFAR10 IMDB
Benchmarks: MNIST CIFAR10 IMDB
Metrics: NC1 NC2A NC2B NC3
Builds on: unconstrained features model neural collapse ResNet with LayerNorm
Closest papers by meaning
- Are All Losses Created Equal: A Neural Collapse Perspective (NeurIPS 2022 · similarity 0.90)
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.90)
- Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data (NeurIPS 2025 · similarity 0.90)
- The Persistence of Neural Collapse Despite Low-Rank Bias (NeurIPS 2025 · similarity 0.89)
- Neural collapse vs. low-rank bias: Is deep neural collapse really optimal? (NeurIPS 2024 · similarity 0.88)
- Generalized Neural Collapse for a Large Number of Classes (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.