Research map / Label-efficient and robust learning / Recognition and long-tail
Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses grokking as a probe to test whether neural collapse (NCC) or relative flatness is necessary for generalization, finding flatness tracks and can be used to induce delayed generalization while NC is neither necessary nor sufficient, and proves collapse implies relative flatness under classical NC assumptions.
Problem
Whether neural collapse and loss-landscape flatness are causal prerequisites for generalization or merely by-products of training dynamics.
Approach
Separates generalization from memorization using grokking, measures NCC and penultimate-layer relative flatness (closed-form bound) over training, and perturbs each quantity via regularization (NCC penalty vs. relative flatness penalty) to test necessity; adds a theorem that NC implies exponentially decaying relative flatness bound.
Key results
- Relative flatness drops sharply at generalization onset in grokking, while NCC already decreases during memorization.
- NCC regularization on CIFAR-10 ResNet-18 leaves validation accuracy unaffected while NCC steadily increases, showing generalization without collapse.
- Relative-flatness regularization delays generalization: on CIFAR-10 final validation accuracy stays around 60% vs ~78% baseline, and on ImageNet-100 around 38% vs ~43%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| discourage neural-collapse clustering with NCC regularization | ResNet-18 · CIFAR-10 | validation accuracy | No clear effect |
| maximize relative flatness during training | ResNet-18 · CIFAR-10 | validation accuracy: around 78% -> about 60% | Hurt |
| remove relative-flatness regularization after epoch 200 | ResNet-18 · CIFAR-10 | validation accuracy: about 60% -> toward 78% | Helped |
| maximize relative flatness throughout training | ViT · ImageNet-100 | validation accuracy: around 38% vs around 43% | Hurt |
| discourage neural-collapse clustering with NCC regularization | ResNet-18 · CIFAR-10 | relative flatness | No clear effect |
| increase NCC regularization strength to 1 × 10^{-2} (ablation) | ResNet-18 · CIFAR-10 | validation accuracy: below 60% | Hurt |
| use NCC regularization strength smaller than 1 × 10^{-3} (ablation) | ResNet-18 · CIFAR-10 | performance metrics | No clear effect |
| add data augmentation with RandomCrop and RandomHorizontalFlip (ablation) | ResNet-18 with NCC regularization · CIFAR-10 | validation accuracy: 78% -> 94% | Helped |
| use NCC regularization strength 0.5 or 0.1 (ablation) | ResNet-18 · CIFAR-10 | accuracy | Hurt |
| use NCC regularization strength 10^{-3} with data augmentation (ablation) | ResNet-18 · CIFAR-10 | NCC | Mixed |
| use relative-flatness regularization strength 10^{-3} (ablation) | ResNet-18 · CIFAR-10 | validation accuracy: slightly below the 78% baseline | No clear effect |
| use relative-flatness regularization strength 10^{-4} (ablation) | ResNet-18 · CIFAR-10 | relative flatness | No clear effect |
Surprising findings
- Penalizing relative flatness induces grokking-like delayed generalization even in ResNets on CIFAR-10 and ViTs on ImageNet-100, where grokking does not typically occur.
- Encouraging flatter minima does not substantially improve generalization, whereas SAM does not reliably find flatter minima.
Limitations
- Necessity is shown empirically under mild assumptions (locally constant labels, representative features) that may fail in other domains.
- Regularized relative-flatness training is unstable and needed weight capping, feature normalization and an unplugging schedule; results sensitive to lambda and seed.
- Focused on classification and small-scale NLP; unclear whether results extend to contrastive pretraining or large LLMs.
Experimental setup
Datasets: CIFAR-10 ImageNet-100 SST-5 modular arithmetic
Benchmarks: CIFAR-10 ImageNet-100 SST-5 modular arithmetic
Metrics: train accuracy validation accuracy relative flatness NCC
Baselines: standard training without regularization
Builds on: relative flatness (Petzka et al.) neural collapse (Papyan et al.) grokking (Power et al.) NCC measure (Galanti et al.)
Closest papers by meaning
- Grokking at the Edge of Numerical Stability (ICLR 2025 · similarity 0.91)
- Grokking as the transition from lazy to rich training dynamics (ICLR 2024 · similarity 0.90)
- Grokking at the Edge of Linear Separability (ICML 2025 · similarity 0.88)
- Deep Networks Always Grok and Here is Why (ICML 2024 · similarity 0.88)
- Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding (ICLR 2024 · similarity 0.88)
- The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.