Research map / Label-efficient and robust learning / Recognition and long-tail
The Geometric Origin of Grokking: Accelerating Generalization via Active Structural Reorganization
Kefei Tao, Zhang Zhang, Mingze Qi, Xiaojun Duan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add R2G loss | transformer baseline · P=500 modular addition | runs reaching full grokking: +73% | Improved |
| add R2G loss | transformer baseline · P=499 modular addition | grokking rate: +80% | Improved |
| add R2G loss | transformer baseline · P=113 modular multiplication | grokking rate: +67% | Improved |
| add R2G loss | 2-layer decoder-only Transformer · P=113 modular addition | grokking rate: +80% | Improved |
| add R2G loss | transformer baseline · Tense-Inflection | grokking success rate: +67% | Improved |
| use R2G instead of Grokfast | P=500 modular addition · P=500 modular addition | grokking success rate: 73% vs 60% | Improved |
| use R2G instead of NeuralGrok | P=500 modular addition · P=500 modular addition | grokking success rate: 73% vs 27% | Improved |
| add angular term to norm-only loss (full R2G) (ablation) | R2G loss · non-grokking datasets | grokking ratio: 45% -> 65% | Improved |
| add norm term to angular-only loss (full R2G) (ablation) | R2G loss · non-grokking datasets | grokking ratio: 55% -> 65%; mean epoch 2500 -> 1882 | Improved |
| use angular-only instead of norm-only loss (ablation) | decoupled R2G variants · non-grokking datasets | grokking ratio: 45% -> 55% | Improved |
| use reciprocal R2G form instead of LayerNorm objective (ablation) | R2G loss · P=500 modular addition | grokking ratio: 60% -> 73% | Improved |
| use reciprocal R2G form instead of Hyperball objective (ablation) | R2G loss · P=500 modular addition | grokking ratio: 33% -> 73% | Improved |
Closest papers by meaning
- Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking (NeurIPS 2025 · similarity 0.91)
- Grokking at the Edge of Numerical Stability (ICLR 2025 · similarity 0.90)
- The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology (NeurIPS 2026 · similarity 0.89)
- Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Model (ICLR 2025 · similarity 0.89)
- Grokking as the transition from lazy to rich training dynamics (ICLR 2024 · similarity 0.89)
- Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
