Research map / Optimization / Implicit bias and generalization
Implicit Bias in State Space Models and Linear Autoregressive Training
Gal Katzhendler, Gal Vardi, Gilad Yehudai
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical study of the implicit bias of linear state space models and linear autoregressive training, where shared parameters are reused across a sample-dependent number of steps.
Problem
Linear SSMs and autoregressive models reuse the same parameters across a sample-dependent number of steps, so different examples constrain shared weights through different predictors; the implicit bias of training under this structure is not well understood.
Approach
Models the setting as multiple non-homogeneous polynomial predictors sharing parameters, and analyzes which solutions interpolating training data are selected under standard interpolation conditions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train linear autoregressive model with gradient descent on exponential loss (synthetic teacher-student, T in {1,3,5}) | Linear autoregressive model · Synthetic teacher-student data | training loss | Improved |
| examples with lower effective degree vs higher effective degree under GD (ablation) | Linear autoregressive model · Synthetic teacher-student data | normalized minimum margin | Improved |
| increase number of autoregressive steps or sequence length per example | Linear AR models and SSMs · Theory (effective-degree max-margin) | influence on limiting classifier | Hurt |
Closest papers by meaning
- Tuning Frequency Bias of State Space Models (ICLR 2025 · similarity 0.87)
- The Implicit Bias of Structured State Space Models Can Be Poisoned With Clean Labels (NeurIPS 2025 · similarity 0.86)
- Autoregressive Learning in Joint KL: Sharp Oracle Bounds and Lower Bounds (NeurIPS 2026 · similarity 0.86)
- From Generalization Analysis to Optimization Designs for State Space Models (ICML 2024 · similarity 0.85)
- The Bias-Variance Tradeoff in Data-Driven Optimization: A Local Misspecification Perspective (NeurIPS 2025 · similarity 0.85)
- Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
