Research map / Label-efficient and robust learning / Recognition and long-tail
Towards Understanding Grokking: An Effective Theory of Representation Learning
Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J Michaud, Max Tegmark, Mike Williams
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add weight decay to the decoder only (ablation) | transformer on modular addition (p=53) · modular addition | generalization speed | Improved |
| Add dropout in the decoder blocks (ablation) | transformer on modular addition (p=53) · modular addition | steps to generalize: under 10^3 steps | Improved |
| Add decoder weight decay (ablation) | toy encoder-decoder model · toy addition | generalization phase | Mixed |
| Make representation learning faster relative to the decoder (ablation) | toy encoder-decoder model · toy addition | generalization phase | Improved |
| Increase training data fraction above critical value | toy addition model · toy addition (p=10) | grokking steps: critical fraction 0.4 | Improved |
Closest papers by meaning
- Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking (NeurIPS 2025 · similarity 0.90)
- Grokking as a First Order Phase Transition in Two Layer Networks (ICLR 2024 · similarity 0.89)
- The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology (NeurIPS 2026 · similarity 0.88)
- Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds (NeurIPS 2026 · similarity 0.88)
- Grokking as the transition from lazy to rich training dynamics (ICLR 2024 · similarity 0.88)
- Grokking at the Edge of Linear Separability (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
