Research map / Learning theory and algorithms / Kernels and regression theory
Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that grokking can arise in homogeneous neural nets trained with large initialization and small weight decay, via a sharp transition from an early kernel-regime (NTK max-margin) predictor to a late rich-regime (KKT max-margin/min-norm) predictor, with diagonal linear net and matrix completion examples.
Problem
Grokking (delayed sharp transition from memorization to generalization) has lacked rigorous proof in neural network settings and a quantitative explanation for why the transition is sharp.
Approach
Large initialization induces an early kernel-regime implicit bias toward NTK predictors, while small weight decay induces a late-phase bias toward min-norm/max-margin predictors; the time-scale separation (1-c)/λ log α vs (1+c)/λ log α makes the transition provably sharp. Also introduces misgrokking, where the order is reversed.
Key results
- Diagonal linear nets on 3-sparse classification (n=256, d=1e5) show grokking: training fits quickly but test accuracy jumps to 100% only in the late phase.
- Misgrokking example (gamma=25, n=32) shows test accuracy first reaching 100% then dropping to nearly 50% with longer training.
- Matrix completion on the 97x97 multiplication table (5% observed) shows training loss minimized in ~10^1 steps while the sharp test-loss transition occurs only after ~10^6 steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the initialization scale (ablation) | Two-layer ReLU net on modular addition · Modular addition test | time to sharp test-accuracy transition: alpha increases from 1 to 100 | Hurt |
| Reduce weight decay (ablation) | Two-layer ReLU net on modular addition · Modular addition test | time to sharp test-accuracy transition | Hurt |
| Increase weight decay (ablation) | Two-layer ReLU net on modular addition · Modular addition test | time to test-accuracy transition | Helped |
| Increase weight decay beyond a small threshold (ablation) | Two-layer ReLU net on modular addition · Modular addition training | training accuracy | Hurt |
| Remove weight decay (ablation) | Two-layer ReLU net on modular addition · Modular addition test | accuracy: test accuracy slowly transitions from ~0\% to 100\% as t increases from 10^{10^2} to 10^{10^4}; with weight decay, the s | Mixed |
| Use large initialization and small non-zero weight decay | Homogeneous neural nets · Classification and regression tasks | transition from kernel predictor to min-norm/max-margin predictor: \frac{1-c}{\lambda} \log \alpha to \frac{1+c}{\lambda} \log \alpha | Helped |
| Train longer, moving from the kernel regime to the rich regime | Diagonal linear net on sparse linear classification · Sparse linear classification test | accuracy: 100\% | Helped |
| Train longer, moving from the early phase to the late phase | Diagonal linear net on large-L2-margin classification · Large-L2-margin linear classification test | accuracy: 100\% to nearly 50\% | Hurt |
| Train longer, moving from the kernel phase to the rich phase | Matrix factorization on the multiplication table · Multiplication-table matrix completion test loss | test loss: \sim 10^1 steps to minimize the training loss; sharp transition in test loss after \sim 10^6 steps | Helped |
Surprising findings
- Without weight decay, grokking still occurs for modular addition but the transition is no longer sharp, taking exponentially longer.
- Misgrokking: a network can reach perfect test accuracy early and then lose it with longer training.
Limitations
- Hmm.
Experimental setup
Datasets: modular addition over Z_97 (synthetic) synthetic sparse linear classification data synthetic Gaussian misgrokking data synthetic multiplication-table matrix completion
Metrics: test accuracy test loss training accuracy
Baselines: NTK predictor max L1-margin classifier max L2-margin classifier nuclear norm minimization
Builds on: Neural tangent kernel Implicit bias of gradient descent on homogeneous networks Power et al. 2022 grokking
Closest papers by meaning
- Grokking as a First Order Phase Transition in Two Layer Networks (ICLR 2024 · similarity 0.88)
- Grokking as the transition from lazy to rich training dynamics (ICLR 2024 · similarity 0.88)
- Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding (ICLR 2024 · similarity 0.87)
- Grokking at the Edge of Numerical Stability (ICLR 2025 · similarity 0.87)
- Deep Networks Always Grok and Here is Why (ICML 2024 · similarity 0.87)
- Grokking at the Edge of Linear Separability (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.