Research map / Optimization / Implicit bias and generalization
Grokking at the Edge of Linear Separability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows grokking arises in a minimal binary logistic regression task when the data is nearly linearly separable from the origin (lambda=d/N near 1/2) with large feature noise, explained via critical slowing down and a 1D solvable model.
Problem
Explaining why grokking (delayed generalization with non-monotonic test loss) occurs, in a minimal solvable model where memorizing and generalizing solutions are well defined.
Approach
Grokking in logistic regression on Gaussian data with a common label arises near the critical ratio lambda=d/N=1/2, where linear separability from the origin changes (via Wendel's theorem); near this point the flat directions cause GD to linger near the overfitting solution, a critical-slowing-down mechanism, analyzed through conformal time and a 2-point 1D model.
Key results
- Grokking (delayed generalization, non-monotonic test loss) appears in the Gaussian logistic model only as lambda approaches 1/2 from below and sigma is large; the limiting norm of S diverges logarithmically as lambda->1/2.
- The simplified 1D two-point model gives S_inf = log(1-2lambda)/(2(lambda-1)), and grokking time difference in conformal time scales as (1/2)log^2(1-2lambda).
- Wendel's theorem implies the data is separable from the origin with probability 1 for lambda>1/2 and inseparable for lambda<1/2 in the large N,d limit.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Set λ below but close to 1/2, with high perpendicular noise σ | Gradient descent logistic regression · Two-Gaussian binary classification | generalization accuracy over training time | Mixed |
| Move λ toward 1/2 from below | Gradient descent logistic regression · Gaussian binary classification | ‖S∞‖ | Hurt |
| Use λ < 1/2 rather than λ > 1/2 | Gradient descent logistic regression · Gaussian binary classification | asymptotic generalization accuracy | Helped |
| Increase perpendicular noise σ | Gradient descent logistic regression · Gaussian binary classification near λ=1/2 from below | grokking time | Mixed |
| Use Adam instead of gradient descent | Logistic regression grokking setup · λ=0.495, σ=1 | grokking | Helped |
| Increase separation-axis variance σ_A from 0 to 0.05 or 0.1 | Two-Gaussian binary classification · λ=0.4, μ=0.2, σ_B=1 | limiting accuracy | Hurt |
| Increase separation-axis variance σ_A from 0 to 0.05 or 0.1 | Two-Gaussian binary classification · λ=0.4, μ=0.2, σ_B=1 | long-time loss | Hurt |
| Set σ_A=0.01 rather than σ_A=0 | Two-Gaussian binary classification · λ=0.4, μ=0.2, σ_B=1 | dynamics relative to σ_A=0 | No clear effect |
Surprising findings
- Adam optimizer yields grokking even at sigma=1, and with accuracy at chance level initially, similar to Power et al.
- Grokking is not explained by regularization, numerical precision, or slingshot mechanisms; it occurs in plain GD on a linear model.
Limitations
- Analysis is in the gradient-flow limit; finite learning-rate effects studied only empirically
- Restricted to linear logistic regression on Gaussian data; nonlinear models and other grokking tasks not analyzed
- Connection to general grokking (e.g., modular arithmetic) is conjectural
Experimental setup
Datasets: synthetic Gaussian data
Metrics: test loss generalization accuracy grokking time
Builds on: Power et al. grokking Soudry et al. implicit bias of gradient descent Wendel's theorem Lyu and Li / Ji and Telgarsky separability framework
Closest papers by meaning
- Grokking at the Edge of Numerical Stability (ICLR 2025 · similarity 0.92)
- Grokking as the transition from lazy to rich training dynamics (ICLR 2024 · similarity 0.91)
- Deep Networks Always Grok and Here is Why (ICML 2024 · similarity 0.90)
- Grokking Beyond the Euclidean Norm of Model Parameters (ICML 2025 · similarity 0.89)
- Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding (ICLR 2024 · similarity 0.89)
- Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.