Research map / Learning theory and algorithms / Kernels and regression theory
Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analytically solves gradient-flow training of linear teacher-student networks with MSE classification, showing grokking arises from generalization loss decaying slower than training loss and is an artifact of the accuracy measure.
Problem
Explaining why grokking (delayed generalization long after training fit) occurs, in the simplest solvable setting of linear networks on linear tasks with Gaussian inputs.
Approach
Derives exact gradient-flow training and generalization loss dynamics for a linear teacher-student model using random matrix theory (Marchenko-Pastur eigenvalue distribution of the training Gram matrix), mapping loss to threshold accuracy via an Erf/chi-square formula, and shows grokking time is governed mainly by lambda = d_in/N_tr.
Key results
- For lambda<1 the grokking time difference approximates log(1/(1-sqrt(lambda)))/(2 eta_0 (1-sqrt(lambda))^2), diverging near lambda=1 and vanishing as lambda approaches 0.
- Grokking time as a function of output dimension is non-monotonic, peaking at a critical d_out (about 50 in the experiments with d_in=1000).
- Analytical predictions for losses and accuracies match numerical experiments with no fitting parameters in under- and overparameterized regimes; 2-layer linear and tanh MLPs are roughly matched by the rescaled learning-rate theory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the input-dimension-to-training-sample ratio λ toward 1 | Linear estimator · Gaussian teacher-student setup | grokking time: maximal near λ ≃ 1; vanishes for λ ≃ 0 | Helped |
| Rescale the student weight initialization | Linear estimator · Gaussian teacher-student setup | grokking time: \mathcal{L}\to \frac{1+\alpha^2}{2} \mathcal{L} | Hurt |
| Increase output dimension d_out | Linear estimator · Gaussian teacher-student setup | grokking time: d_\mathrm{out}^\mathrm{max}\simeq 50 | Mixed |
| Add L2 weight decay in the overparameterized regime | Linear estimator · Gaussian teacher-student setup, λ > 1 | grokking time: 1/\gamma | Hurt |
| Add L2 weight decay in the underparameterized regime | Linear estimator · Gaussian teacher-student setup, λ < 1 | grokking time: \Delta t_\mathrm{grok} \simeq \tfrac{\log (1 + \sqrt{\lambda })}{2 \eta _0 (1-\sqrt{\lambda })^2} | Mixed |
| Add training label noise | Linear estimator · Gaussian teacher-student setup | generalization accuracy: \sigma^2_\delta | Hurt |
| Use a deeper network with one hidden layer and linear activations | Linear estimator · Gaussian teacher-student setup | training and generalization dynamics | No clear effect |
| Use tanh activations in a two-layer MLP | Linear estimator · Gaussian teacher-student setup | training and generalization dynamics | No clear effect |
Surprising findings
- Grokking here reflects only that generalization loss decays slower than training loss; nothing qualitative changes at the grokking point, so it is an artifact of the accuracy threshold rather than a memorization-to-understanding transition.
- Increasing output dimension can reduce grokking time at large d_out because accuracy saturates near 100% regardless of loss.
Limitations
- Exact predictions hold only in the gradient-flow limit and for large input dimension and sample size.
- Multilayer and nonlinear results are semi-analytic or preliminary, based on NTK-like approximations and small-learning-rate assumptions.
- Uses MSE-based classification accuracy rather than softmax or cross-entropy.
Experimental setup
Datasets: synthetic Gaussian teacher-student data
Metrics: training loss generalization loss training accuracy generalization accuracy grokking time
Builds on: Marchenko-Pastur law neural tangent kernel grokking (Power et al.)
Closest papers by meaning
- Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Model (ICLR 2025 · similarity 0.91)
- Grokking as the transition from lazy to rich training dynamics (ICLR 2024 · similarity 0.91)
- To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets (ICLR 2024 · similarity 0.90)
- Deep Networks Always Grok and Here is Why (ICML 2024 · similarity 0.90)
- Grokking at the Edge of Numerical Stability (ICLR 2025 · similarity 0.89)
- Grokking Beyond the Euclidean Norm of Model Parameters (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.