Research map / Optimization / Implicit bias and generalization
Convergence and Implicit Bias of Gradient Descent on Continual Linear Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that sequential gradient descent on unregularized logistic loss with a fixed per-task budget converges in direction to the joint max-margin solution under cyclic/random task orders, derives non-asymptotic forgetting bounds, and proves convergence to the unique joint minimizer for non-separable tasks.
Problem
Understanding continual learning of multiple linear classification tasks when each task gets only a finite budget of gradient descent iterations, including implicit bias and forgetting, which prior projection-based (Sequential Max-Margin) analyses did not cover.
Approach
Analyzes sequential GD with K unregularized steps per task (instead of training each task to convergence with weak regularization), proving the iterates align with the joint offline max-margin direction, deriving cycle-averaged forgetting bounds tied to pairwise task alignment, and showing convergence to the unique minimizer of the joint loss in the non-separable case.
Key results
- Sequential GD on jointly separable cyclic tasks converges in direction to the joint max-margin solution, with ||w|| growing as O(ln t) (Theorem on directional convergence).
- Cycle-averaged forgetting converges at rate O(ln^4 J / J^2), faster than the O(ln^2 J / J) joint loss bound.
- For non-separable tasks, sequential GD converges to the unique joint minimizer at rate O(ln^2 J / J^2) in squared distance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Repeat jointly separable tasks using sequential GD in cyclic order | Linear classifier trained with unregularized logistic loss · Jointly separable tasks, cyclic order | joint training loss: loss converges to 0 | Helped |
| Repeat jointly separable tasks using sequential GD in cyclic order | Linear classifier trained with unregularized logistic loss · Jointly separable tasks, cyclic order | directional alignment with joint max-margin solution: lim_{t → ∞} w_k^(t)/||w_k^(t)|| = hat{w}/||hat{w}|| | Helped |
| Use independently and uniformly sampled task order instead of cyclic order | Sequential GD · Jointly separable tasks, random order | joint training loss: loss converges to zero almost surely | Helped |
| Use independently and uniformly sampled task order instead of cyclic order | Sequential GD · Jointly separable tasks, random order | directional alignment with joint max-margin solution: w_k^(t) = ln(t) hat{w} + rho_k^(t) | Helped |
| Repeat tasks in cyclic order | Sequential GD · Jointly separable tasks | cycle-averaged forgetting: \gO(\frac{\ln^4 J}{J^2}) | Helped |
| Increase negative alignment between tasks | Sequential GD · Jointly separable tasks, cyclic order | cycle-averaged forgetting | Hurt |
| Increase positive alignment between tasks | Sequential GD · Jointly separable tasks, cyclic order | cycle-averaged forgetting | Helped |
| Train jointly non-separable tasks in cyclic order | Sequential GD · Jointly non-separable tasks | distance to the unique joint-loss minimum: \gO(\tfrac{\ln^2 J}{J^2}) | Helped |
| Resample fresh data from each task distribution at every stage | Sequential GD · Jointly separable online data, cyclic and random orders | directional convergence toward the joint max-margin direction | Helped |
| Use a two-layer ReLU network instead of a linear classifier | Sequential GD · Synthetic tasks, cyclic order | decision-boundary similarity to jointly trained model | Helped |
| Train a linear model on CIFAR-10 tasks in cyclic order | Sequential GD · CIFAR-10, airplane and automobile classes | distance to jointly trained solution: converges close to 0 | Helped |
Surprising findings
- The joint max-margin direction can lie outside the span of individual task max-margin directions, yet sequential GD still converges to it.
- Joint training loss can temporarily increase within a cycle even with a small learning rate.
Limitations
- Results restricted to small learning-rate regime, excluding large learning rates and edge-of-stability.
- Analysis limited to linear models with fixed per-task datasets; neural network results are empirical only.
Experimental setup
Datasets: CIFAR-10
Metrics: angle to joint max-margin direction joint training loss cycle-averaged forgetting distance to joint minimizer
Baselines: Sequential Max-Margin (SMM) joint full-batch GD
Builds on: Sequential Max-Margin (Evron et al., 2022) implicit bias of GD on separable data (Soudry et al., 2018)
Closest papers by meaning
- On the Theory of Continual Learning with Gradient Descent for Neural Networks (ICML 2026 · similarity 0.92)
- An Effective Dynamic Gradient Calibration Method for Continual Learning (ICML 2024 · similarity 0.89)
- Optimal Rates in Continual Linear Regression via Increasing Regularization (NeurIPS 2025 · similarity 0.89)
- Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning (ICLR 2024 · similarity 0.88)
- Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers (ICML 2025 · similarity 0.88)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.