Research map / Optimization / Implicit bias and generalization
Grokking phase transitions in learning local rules with gradient descent
Bojan Žunkovič, Enej Ilievski
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add L1 regularisation instead of L2 (larger grokking probability, shorter grokking time, smaller effective dimension) (ablation) | D-ball perceptron model · D-ball model, grokking probability | grokking probability: above 90% and always larger than 0.915 for L1; exponentially small in D for lambda1=0 | Improved |
| increase L2 regularisation strength lambda2 (ablation) | D-ball perceptron model · D-ball model, grokking probability | grokking probability: increases up to a maximum depending on other parameters | Improved |
| increase latent dimension D (L2 only) (ablation) | D-ball perceptron model · D-ball model, grokking probability | grokking probability: exponentially decreases with D | Hurt |
| increase number of training samples N and class separation epsilon (ablation) | 1D exponential model · 1D model, grokking probability and time | grokking probability / grokking time: N=2, 5, 10 and epsilon_lambda=0.4, 0.04, 0.004 compared | Improved |
| add L2 regularisation to fixed-attention model (ablation) | Tensor-network attention (fixed A) · Fixed attention, rule 30 (M=3 samples) | grokking probability | Hurt |
| add small L1 regularisation to fixed-attention model (ablation) | Tensor-network attention (fixed A) · Fixed attention, rule 30 (M=3 samples) | grokking probability | Mixed |
| add L1 or L2 regularisation (ablation) | Tensor-network attention (fixed A) · Fixed attention, rule 30, Examples 1-3 | critical exponent nu: nu 0.85, 0.75, 0.84 all below one | No clear effect |
| add regularisation (L1 or L2) to full trained model (ablation) | Tensor-network attention (trained A,B) · Full model, rule 30 | critical exponent nu: 2.5 -> 0.7 (L1), 0.9 (L2) | Improved |
| add regularisation to full trained model (ablation) | Tensor-network attention (trained A,B) · Full model, rule 30 | test error after grokking | Hurt |
| increase bond dimension d of the model (ablation) | Tensor-network attention (trained A,B) · Full model, rule 30 | critical exponent nu / effective dimension | No clear effect |
| increase L2 regularisation in fixed attention model (ablation) | Tensor-network attention (fixed A) · Fixed attention, rule 30 | grokking time | Hurt |
| larger effective dimension D_eff and larger class separation epsilon (ablation) | Tensor-network attention (fixed A) · Fixed attention, rule 30 | grokking time: larger D_eff gives longer grokking times; larger epsilon gives smaller | Mixed |
Closest papers by meaning
- Grokking as a First Order Phase Transition in Two Layer Networks (ICLR 2024 · similarity 0.90)
- Is Grokking a Computational Glass Relaxation? (NeurIPS 2025 · similarity 0.88)
- Grokking at the Edge of Numerical Stability (ICLR 2025 · similarity 0.87)
- Grokking at the Edge of Linear Separability (ICML 2025 · similarity 0.87)
- Grokking as the transition from lazy to rich training dynamics (ICLR 2024 · similarity 0.87)
- Grokking Beyond the Euclidean Norm of Model Parameters (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
