Research map / Optimization / Implicit bias and generalization
Grokking Beyond the Euclidean Norm of Model Parameters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that gradient descent with small non-zero regularization of a generalizing property P (l1, nuclear norm, Sobolev) induces grokking, with delay scaling as 1/(alpha*beta), and that l2 norm is unreliable as a grokking progress measure.
Problem
Grokking (delayed generalization after memorization) is usually explained by l2 weight decay and large initialization; it is unclear whether l2 is needed or whether the l2 norm reliably tracks generalization.
Approach
A general theorem shows that for a loss g satisfying a Chatterjee-Łojasiewicz condition at initialization, subgradient descent on g + beta*h (for any suitable regularizer h such as l1 or nuclear norm) memorizes first and then, after Theta(1/(alpha*beta)) steps, biases toward minimizers of h; if h aligns with the generalizing structure (sparsity, low rank), grokking follows. Depth provides implicit
Key results
- Sparse recovery with l1 regularization (alpha=0.1, beta=1e-5, n=100, s=5, N=30) shows a grokking-like pattern: training error reaches 1e-6 early and recovery error drops much later.
- Generalization delay scales inversely with alpha*beta; l1, l2 and nuclear-norm regularization all show the same effect on modular addition.
- In shallow sparse and low-rank settings, l2 regularization alone gives an abrupt generalization transition that does not reach the optimal solution ('grokking without understanding'), and l2 norm grows without harming generalization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use small non-zero L1 regularization instead of relying on L2 weight decay | Sparse recovery · Sparse recovery | generalization delay | Helped |
| Use nuclear-norm regularization instead of relying on L2 weight decay | Low-rank matrix factorization · Matrix completion | recovery error | Helped |
| Replace L1 or nuclear-norm regularization with L2 regularization (ablation) | Shallow sparse recovery and matrix factorization · Sparse recovery and matrix completion with N < n | generalization | Hurt |
| Increase the learning rate α while holding β fixed | Regularized training · Modular addition and nonlinear teacher-student | generalization delay | Helped |
| Decrease the regularization strength β while holding other factors fixed | Regularized training · Modular addition and algorithmic data | generalization delay | Hurt |
| Add depth to the linear model | Sparse recovery model · Sparse recovery | generalization error | Helped |
| Use a shallow model instead of a deeper over-parameterized model (ablation) | Sparse recovery model · Sparse recovery | generalization | Hurt |
| Select matrix-completion entries with higher coherence scores | Nuclear-norm-regularized matrix completion · Matrix completion | generalization performance | Helped |
| Increase measurement coherence | Compressed sensing · Sparse recovery | generalization delay | Hurt |
| Use lower-coherence measurements | Compressed sensing · Sparse recovery | recovery quality | Helped |
| Replace L2 regularization with L1 or nuclear-norm regularization | MLP training · Modular addition | generalization | No clear effect |
| Use L1 regularization for matrix factorization instead of nuclear-norm regularization (ablation) | Matrix factorization · Low-rank matrix factorization | grokking | Hurt |
Surprising findings
- The l2 norm of parameters increases during training and after generalization in settings without weight decay, yet generalization still occurs.
- Increasing data coherence in matrix factorization/data selection can amplify grokking (faster generalization with fewer samples), while the opposite holds for compressed sensing.
Limitations
- Theory covers subgradient descent under CL-at-initialization assumptions and specific linear settings (sparse recovery, matrix factorization).
- Empirical validation is on small synthetic problems and modular addition/MNIST, not large-scale models.
Experimental setup
Datasets: modular addition (algorithmic data) synthetic sparse recovery data synthetic matrix completion data MNIST
Benchmarks: modular addition (algorithmic) MNIST
Metrics: training accuracy validation accuracy recovery error training error l2 norm of parameters
Baselines: l2 weight decay minimum l2-norm least squares
Builds on: grokking (Power et al. 2022) Chatterjee-Łojasiewicz inequality Lasso / compressed sensing deep matrix factorization implicit regularization
Closest papers by meaning
- Grokking at the Edge of Numerical Stability (ICLR 2025 · similarity 0.93)
- Why Do You Grok? A Theoretical Analysis on Grokking Modular Addition (ICML 2024 · similarity 0.91)
- Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Model (ICLR 2025 · similarity 0.90)
- Deep Networks Always Grok and Here is Why (ICML 2024 · similarity 0.90)
- A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisation (NeurIPS 2025 · similarity 0.90)
- Grokking at the Edge of Linear Separability (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.