Research map / Adversarial robustness and security / Certified and empirical robustness
Deep Networks Always Grok and Here is Why
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that grokking (delayed generalization and a new "delayed robustness" to adversarial examples) occurs broadly in standard training, and explains it via a spline-based local complexity progress measure whose second descent ("region migration") precedes grokking.
Problem
Grokking (delayed generalization) was thought to occur only in controlled settings such as large-norm initialization or algorithmic transformer tasks; it is unclear how widespread it is and what training dynamics cause it, including whether robustness to adversarial examples can be grokked too.
Approach
Introduces "delayed robustness" (grokking of PGD adversarial examples) and a task-agnostic local complexity measure, counting neuron hyperplanes intersecting a cross-polytope neighborhood around data points in the spline partition view of DNNs. Training shows three phases (descent, ascent, second descent/region migration), with region migration moving linear regions from training points toward the
Key results
- Delayed robustness observed across MLP-MNIST, CNN-CIFAR10/100, ResNet18 on CIFAR10 and Imagenette, and a GPT on Shakespeare text, with adversarial accuracy rising long after test accuracy converges (e.g., ResNet18 on CIFAR10 starts grokking adversarial examples after about 10^4 steps and reaches nea
- Local complexity shows three phases (first descent, ascent, second descent/region migration), and the second descent always precedes the onset of delayed generalization or robustness in the experiments.
- Batch normalization removes grokking of adversarial examples; increasing training data size expedites grokking while increasing random-label memorization delays region migration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase batch size from 64 to 512 | ResNet18 · CIFAR10 | adversarial grokking onset | Helped |
| Increase depth from 2 to 6 | MLP · MNIST | adversarial grokking onset | Helped |
| Increase layers from 6 to 12 | GPT · Shakespeare Text | adversarial grokking onset | Helped |
| Increase width from 20 to 2000 | MLP · MNIST | adversarial grokking onset | Helped |
| Increase width from 16 to 64 filters | ResNet18 · CIFAR10 | adversarial grokking onset | Helped |
| Increase weight decay from 0 to 1 | MLP · MNIST | adversarial grokking onset | Hurt |
| Increase training data from 1K to 60K | MLP · MNIST | adversarial grokking onset | Helped |
| Increase randomly labeled training data from 1K to 60K | MLP · MNIST, randomly labeled | adversarial grokking onset | Hurt |
| Add BatchNorm after convolution | ResNet18 · CIFAR10 | adversarial grokking occurrence | Hurt |
| Add BatchNorm after convolution | CNN · CIFAR10 | adversarial grokking occurrence | Hurt |
| Vary weight decay (ablation) | CNN, depth 5 and width 32 · CIFAR10 | region migration onset | Mixed |
| Use GeLU activation instead of ReLU (ablation) | MLP and Transformer · MNIST and Shakespeare Text | local complexity training dynamics | No clear effect |
Surprising findings
- Batch normalization removes the first descent and prevents adversarial grokking, with local complexity increasing monotonically on ImageNet-scale ResNet18 training.
- Weight decay has non-monotonic effects: it both delays and hastens region migration depending on the amount.
Limitations
- Theoretical justification for the double descent in local complexity and for why region migration happens late in training is lacking.
- Local complexity is an approximation (counting hyperplanes intersecting a cross-polytope), which can deform across layers for larger radii.
- Authors note the mechanism for the transition from accumulation to repelling of non-linearities around training data is not understood.
Experimental setup
Datasets: MNIST CIFAR10 CIFAR100 Imagenette Imagenet Shakespeare Text IMDb modular addition
Benchmarks: MNIST CIFAR10 CIFAR100 Imagenette Shakespeare Text
Metrics: test accuracy adversarial accuracy under l_inf PGD local complexity AutoAttack robustness
Baselines: AutoAttack
Builds on: grokking (Power et al.) affine spline view of deep networks (Balestriero & Baraniuk) progress measures for grokking (Nanda et al.) PGD adversarial attacks SplineCam
Closest papers by meaning
- Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Model (ICLR 2025 · similarity 0.92)
- Grokking at the Edge of Numerical Stability (ICLR 2025 · similarity 0.92)
- Grokking as the transition from lazy to rich training dynamics (ICLR 2024 · similarity 0.91)
- To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets (ICLR 2024 · similarity 0.90)
- Grokking Beyond the Euclidean Norm of Model Parameters (ICML 2025 · similarity 0.90)
- Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.