Research map / Optimization / Implicit bias and generalization
Improving Adaptivity via Over-Parameterization in Sequence Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that eigenvalue ordering matters for kernel regression and proposes over-parameterized (learnable-eigenvalue) gradient flow in a Gaussian sequence model, proving adaptive near-optimal rates with a universal stopping time and benefits of depth.
Problem
Kernel regression with a fixed kernel performs poorly when the ordering of kernel eigenvalues is misaligned with the coefficients of the target function in the eigenbasis, so the best kernel depends on unknown signal structure.
Approach
Replace the fixed square-root eigenvalues in a diagonalized kernel gradient flow with trainable per-component parameters (theta_j = a_j b_j^D beta_j), initialized at the eigenvalues, so that gradient descent adapts the effective eigenvalues to the signal; deeper factors further refine this adaptation.
Key results
- Over-parameterized GD attains near-optimal rate epsilon^{2p/(p+1)} (up to log factor) versus epsilon^{2p/(p+q)} for vanilla GF, with a universal stopping time t ~ epsilon^{-(2D+2)/(D+2)}.
- Simulations give fitted convergence rates close to theory (about 0.5 for OpGD vs 0.33 for vanilla GD in the p=1, q=2 setting).
- Depth D=1,2 further reduces the error for small-eigenvalue components, e.g. faster rates at gamma=1.1 for D=2 than D=0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn eigenvalues with over-parameterized gradient descent instead of using fixed eigenvalues | Sequence model · Sequence model (p=1, q=2) | convergence rate: 0.33 -> 0.5 | Helped |
| add one extra parameterization layer (D=1) (ablation) | OpGD · Sequence model (p=1, q=2, gamma=1.1) | convergence rate: 0.48 -> 0.49 | Helped |
| add one extra parameterization layer (D=1) (ablation) | OpGD · Sequence model (p=1, q=2, gamma=2) | convergence rate: 0.50 -> 0.50 | No clear effect |
| add one extra parameterization layer (D=1) (ablation) | OpGD · Sequence model (p=1, q=2, gamma=3) | convergence rate: 0.50 -> 0.44 | Hurt |
| add two extra parameterization layers (D=2) (ablation) | OpGD · Sequence model (p=1, q=2, gamma=1.1) | convergence rate: 0.48 -> 0.48 | No clear effect |
| add two extra parameterization layers (D=2) (ablation) | OpGD · Sequence model (p=1, q=2, gamma=2) | convergence rate: 0.50 -> 0.49 | Hurt |
| add two extra parameterization layers (D=2) (ablation) | OpGD · Sequence model (p=1, q=2, gamma=3) | convergence rate: 0.50 -> 0.50 | No clear effect |
| increase parameterization depth (ablation) | Over-parameterized gradient descent · Sequence model, gamma=1.1 | convergence rate: 0.38 -> 0.34 | Mixed |
Limitations
- Analysis restricted to diagonal sequence model and gradient flow; the coupled kernel-regression version is not analyzed.
- Eigenfunctions are kept fixed; learning eigenfunctions is left open.
- Extra epsilon^{-2/(D+2)/gamma} term remains for small gamma, and a log gap separates signal and noise components.
Experimental setup
Datasets: California Housing Concrete Compressive Strength
Benchmarks: California Housing Concrete Compressive Strength
Metrics: generalization error convergence rate exponent
Baselines: vanilla gradient flow fixed-eigenvalue kernel gradient descent
Builds on: neural tangent kernel diagonal linear networks implicit regularization Mercer decomposition
Closest papers by meaning
- Generalization in Kernel Regression Under Realistic Assumptions (ICML 2024 · similarity 0.90)
- Loss Landscape Characterization of Neural Networks without Over-Parametrization (NeurIPS 2024 · similarity 0.87)
- Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel (NeurIPS 2025 · similarity 0.87)
- Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit (NeurIPS 2026 · similarity 0.86)
- Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift (ICML 2024 · similarity 0.86)
- A Theoretical Framework For Overfitting In Energy-based Modeling (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.