Research map / Architectures and efficiency / RNNs and normalization
Slower Generalization, Faster Memorization: A Sweet Spot in Algorithmic Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that in Needleman-Wunsch matrix generation, small decoder-only Transformers reach high validation exact-match fastest (in gradient updates) at an intermediate dataset size, a "sweet spot" absent in three-digit multiplication.
Problem
Critical-data-size grokking accounts explain when generalization becomes possible, but not which dataset size gives the fastest update-based validation convergence after generalization is attainable, especially for structured-output tasks.
Approach
Dataset-size sweeps of train and validation exact-match threshold-crossing times in a structured dynamic-programming output task (Needleman-Wunsch matrix generation) reveal an interior optimum, interpreted as rule discovery saturating while residual exact-fitting cost grows; a random-suffix probe and train-random gap diagnostic support this two-pressure account.
Key results
- NW validation crossing time at strict threshold 0.98 is non-monotonic in dataset size, with an interior sweet spot (N_c < N* < N_max), while multiplication shows no pronounced post-threshold slowdown.
- In the partial-generalization regime, increasing N can reduce the number of updates needed to reach high training accuracy in NW.
- In a random-suffix probe, the rule-generated NW matrix is fit before the much shorter random suffix at larger N.
Surprising findings
- Larger datasets can require more optimizer updates to reach high validation accuracy, even though they still generalize, reversing the usual post-threshold intuition.
- Epoch-normalized analyses recover the conventional view that larger datasets need fewer passes, so the slowdown is specific to update-based convergence.
Limitations
- Results limited to small Transformers, a single sequence length family (L=5, with L=4 replication), and a finite synthetic task universe; external validity to natural-language reasoning is not claimed.
- The two-pressure mechanism is an empirical explanation; the random-suffix probe does not identify internal circuits.
- Non-crossing runs are censored, so the sweet spot is defined only among dataset sizes reaching high validation accuracy.
Experimental setup
Datasets: Needleman-Wunsch matrix generation (synthetic, L=5, alphabet size 4) three-digit multiplication (synthetic) decimal addition (synthetic)
Benchmarks: Needleman-Wunsch matrix generation (L=5, L=4) three-digit multiplication decimal addition
Metrics: exact-match accuracy optimizer updates to threshold crossing epochs to threshold train-random accuracy gap
Baselines: three-digit multiplication decimal addition
Builds on: grokking critical data size AdamW
Closest papers by meaning
- The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology (NeurIPS 2026 · similarity 0.89)
- To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets (ICLR 2024 · similarity 0.89)
- Transformers Converge to Invariant Algorithmic Cores (NeurIPS 2026 · similarity 0.88)
- Grokking at the Edge of Numerical Stability (ICLR 2025 · similarity 0.87)
- Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds (NeurIPS 2026 · similarity 0.87)
- What Do Learning Dynamics Reveal About Generalization in LLM Mathematical Reasoning? (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.