Research map / Learning theory and algorithms / Sample complexity

Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds

NeurIPS 2026 · Learning theory and algorithms › Sample complexity · approach: PAC learnability theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes that grokking onset on modular division is set by where memorisation-speed and generalisation-speed curves intersect as a function of Transformer parameter count, using bits-per-parameter capacity estimates and random-label memorisation timing.

Problem

Grokking's dependence on model size is not explained: small models generalise immediately and grokking onset lies above the capacity threshold where memorisation first becomes representable. The paper asks what sets the parameter scale at which grokking begins.

Approach

Model grokking as a race between two measurable timescales: memorisation speed T_mem(P) (epochs to fit random labels of equal bit content) and generalisation speed T_gen(P) (epochs to reach high validation accuracy on modular division). Grokking onset is predicted by the parameter count where these curves cross, and T_mem is empirically tied to the capacity fraction f = K/(C_model P).

Key results

• Predicted intersection-based grokking onset rank-correlates with empirical onset (Spearman rho=0.976 on the central sweep, log-log slope 0.985).
• Capacity fit gives C_model approx 2.16 bits per parameter for the Transformer family.
• Memorisation time T_mem fits an exponential in capacity fraction f for f in [0, 0.25], collapsing curves across datasets.

Surprising findings

• Grokking onset lies strictly above the capacity threshold P_mem; models above P_mem still generalise immediately.
• Empirical onset arrives roughly 30% earlier than the intersection prediction, a constant offset across primes.

Limitations

• T_gen(P) is measured rather than derived, so new architectures or tasks require re-running speed measurements.
• Calibration constant shifts with weight decay, depth, and training fraction; cross-setting prediction needs recalibration.
• Only small decoder-only Transformers on modular division and addition are studied; the exponential fit is empirical.

Experimental setup

Datasets: modular division over Z_p (primes 97-139) random-label synthetic datasets

Benchmarks: modular division grokking onset across primes

Metrics: epochs to saturation (T_mem, T_gen) generalisation delay Spearman rho Lin's CCC nested OLS F-test bits per parameter

Baselines: bits-per-parameter capacity threshold P_mem circuit efficiency account pattern-learning speed account

Builds on: bits-per-parameter capacity framework (Morris et al.) grokking (Power et al.) AdamW

Closest papers by meaning

• The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology (NeurIPS 2026 · similarity 0.89)
• Emergence in non-neural models: grokking modular arithmetic via average gradient outer product (ICML 2025 · similarity 0.87)
• Slower Generalization, Faster Memorization: A Sweet Spot in Algorithmic Learning (NeurIPS 2026 · similarity 0.87)
• Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding (ICLR 2024 · similarity 0.87)
• Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks (NeurIPS 2024 · similarity 0.87)
• Is Grokking a Computational Glass Relaxation? (NeurIPS 2025 · similarity 0.86)