Research map / Optimization / Implicit bias and generalization
Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a telescoping model that expresses a trained network as a sum of linearized per-step functional updates, and uses it to analyze double descent, grokking, gradient boosting vs. neural nets on tabular data, and linear mode connectivity.
Problem
Neural networks often behave unexpectedly (non-monotonic generalization, grokking, tabular underperformance vs. GBTs, weight-averaging success), and there is no simple empirical tool to analyze the learned function from its training trajectory.
Approach
Approximate a trained network by telescoping first-order linearizations of each SGD functional update, yielding a weight-averaging and kernel representation f̃ = f0 + sum_t ∇f_{t-1}(x)^T Δθ_t; this makes learned predictions a linear combination of training labels, enabling effective-parameter complexity measures and comparisons with gradient boosting trees.
Key results
- The telescoping model reduces approximation error by orders of magnitude relative to the first-order linearization f^lin around theta_0 for small learning rates.
- Test-time effective parameters p^test decrease past the interpolation threshold while train-time p^train increases monotonically in single-hidden-layer ReLU networks on CIFAR-10 cats vs dogs.
- Neural network performance decays faster than GBTs as test data becomes more irregular, tracked by unbounded tangent kernel norms on the houses dataset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Iteratively telescoping first-order updates instead of a single linear expansion around theta_0 | 3-layer ReLU MLP trained on MNIST 3-vs-5 · MNIST 3-vs-5 approximation error | mean absolute approximation error: by orders of magnitude | Helped |
| Small learning rate gamma in telescoping model | telescoping model approximation · MNIST 3-vs-5 | mean absolute approximation error | Helped |
| Adam(W) instead of SGD requires smaller learning rate for approximation quality | telescoping model with AdamW · MNIST 3-vs-5 | approximation quality | Hurt |
| Binary-classification lower learning rate for grokking replication | grokking experiments (MNIST, polynomial regression) · grokking replication | visual grokking extremeness | Hurt |
| Standard initialization scale with sigmoid output vs large initialization (ablation) | MNIST grokking setup · MNIST generalization speed | steps to generalization: 10^2 instead of 10^5 steps | Helped |
| Adding sigmoid output activation with standard initialization (ablation) | MNIST with standard initialization · effective parameters p^0 on MNIST | learned complexity (effective parameters) | Helped |
| Pre-training on SVHN before weight averaging | ResNet-20 CIFAR-10 linear mode connectivity · LMC loss barrier | loss barrier | Helped |
| Tree kernel (GBT) vs tangent kernel (NN) under increasing test irregularity | neural networks vs gradient boosted trees on tabular data · houses dataset, increasing irregular test proportion | relative MSE of NN vs GBT | No clear effect |
| Momentum added to SGD functional updates | telescoping model · theoretical analysis | functional update sign alignment | Mixed |
Surprising findings
- The telescoping model of a neural network and gradient boosted trees have identical structure and differ only in their kernel.
- Gradient stabilization explains linear mode connectivity only partially: BatchNorm parameter gradients change more after pre-training, so the mechanism cannot fully account for LMC.
Limitations
- Obtaining the telescoping model requires gradients for every train and test example at each step, which is costly and prohibitive for very large networks and datasets.
- Gradient stabilization does not fully explain linear mode connectivity, notably for pre-trained models.
- Case studies are restricted to specific, small empirical examples rather than broad generality.
Experimental setup
Datasets: MNIST CIFAR-10 SVHN tabular benchmark datasets (houses and others) polynomial regression task
Benchmarks: CIFAR-10 MNIST tabular benchmark datasets (houses)
Metrics: effective parameters p^0 (train and test) mean absolute approximation error MSE relative to GBT maximal accuracy decrease under weight averaging squared change in gradients
Baselines: linearized first-order model f^lin gradient boosted trees
Builds on: neural tangent kernel effective parameters of smoothers (Hastie et al.) gradient boosting AdamW deep double descent
Closest papers by meaning
- Grokking as the transition from lazy to rich training dynamics (ICLR 2024 · similarity 0.87)
- Why Do You Grok? A Theoretical Analysis on Grokking Modular Addition (ICML 2024 · similarity 0.86)
- Grokking at the Edge of Numerical Stability (ICLR 2025 · similarity 0.86)
- Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding (ICLR 2024 · similarity 0.86)
- Grokking as a First Order Phase Transition in Two Layer Networks (ICLR 2024 · similarity 0.85)
- Deep Networks Always Grok and Here is Why (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.