Research map / Optimization / Implicit bias and generalization
Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokking
Ali Saheb Pasand, Elvis Dohmatob
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace vanilla SGD with EGD (exact SVD) | two-layer MLP · Modular addition (p=79) | epochs to 95% test accuracy: 10099 -> 222 | Improved |
| replace vanilla SGD with EGD (exact SVD) | two-layer MLP · Modular multiplication (p=97) | epochs to 95% test accuracy: 5715 -> 107 | Improved |
| replace vanilla SGD with EGD (exact SVD) | two-layer MLP · Sparse parity (k=2) | epochs to 95% test accuracy: 7873 -> 109 | Improved |
| use randomized SVD with rank 128 instead of exact SVD (ablation) | EGD · Modular addition (p=127) | epochs to 95% test accuracy: 62 -> 76 epochs; 11.4 -> 10.1 s | Mixed |
| use randomized SVD rank 40 instead of exact SVD (ablation) | EGD · Sparse parity (k=2) | epochs to 95% test accuracy: 109 -> 42 | Improved |
| use randomized SVD rank 40 instead of exact SVD (ablation) | EGD · Sparse parity (k=4) | epochs to 95% test accuracy: 254 -> - | Hurt |
| reduce RSVD rank from 40 to 10 (ablation) | EGD · Sparse parity (k=2) | epochs to 95% test accuracy: 42 -> 2586 | Hurt |
| use RSVD rank 30 instead of exact SVD (ablation) | EGD · Sparse parity (k=3) | epochs to 95% test accuracy: 115 -> 76 | Improved |
| use column normalization (SVD-free) instead of exact SVD (ablation) | EGD · Modular addition (p=127) | epochs to 95% test accuracy: 62 -> 133 epochs; 11.4 -> 5.9 s | Mixed |
| use column normalization instead of exact SVD (ablation) | EGD · Sparse parity (k=4) | epochs to 95% test accuracy: 254 -> - | Hurt |
| use best-rank RSVD instead of exact SVD, average over tasks (ablation) | EGD · Average over all tasks | epochs to 95% test accuracy | Improved |
Closest papers by meaning
- Grokking at the Edge of Numerical Stability (ICLR 2025 · similarity 0.90)
- Why Do You Grok? A Theoretical Analysis on Grokking Modular Addition (ICML 2024 · similarity 0.90)
- Grokking at the Edge of Linear Separability (ICML 2025 · similarity 0.90)
- Grokking Beyond the Euclidean Norm of Model Parameters (ICML 2025 · similarity 0.89)
- Grokking as the transition from lazy to rich training dynamics (ICLR 2024 · similarity 0.89)
- A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisation (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
