Research map / Optimization / Implicit bias and generalization
Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes large-stepsize gradient descent on non-homogeneous two-layer networks with logistic loss, proving a stable phase after risk drops below a stepsize-dependent threshold, margin improvement, and an O(1/T^2) accelerated risk bound.
Problem
Understanding why large-stepsize GD on non-homogeneous neural networks under logistic loss shows an oscillating edge-of-stability phase followed by monotone decrease, and whether that large-stepsize regime improves margin and optimization speed beyond NTK and mean-field analyses.
Approach
Extends implicit-bias and large-stepsize theory from linear and homogeneous predictors to nearly 1-homogeneous (non-homogeneous) two-layer networks by using a bounded homogeneous error assumption and constructing auxiliary margin functions that absorb large-stepsize and non-homogeneity errors; shows the average risk decreases during the oscillatory phase, so the stable phase is reached in finite s
Key results
- Theorem: once empirical risk falls below 1/(eta(2rho^2+beta)), GD decreases risk monotonically (stable phase), and for nearly homogeneous predictors the normalized margin nearly monotonically increases.
- For linearly separable data with bounded derivative, the average risk over the first t steps is O((log^2(eta t)+eta^2)/(eta t)), so the EoS phase ends in finite steps.
- With a suitably large stepsize tuned to budget T, the risk after T steps is O(1/T^2), whereas monotone GD incurs an Omega(1/t) lower bound in the classical regime.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Choose a suitably large stepsize that balances the EoS and stable phases | GD for two-layer networks · Linearly separable datasets | empirical risk: O(1/T^2) vs. Ω(1/t) | Helped |
| Use a larger stepsize | GD for two-layer networks · Two-layer network experiments (XOR and CIFAR-10 subset) | empirical risk | Helped |
| Enter the stable phase | Large-stepsize GD for two-layer networks · Two-layer networks, stable phase | empirical risk: O(1/(η t)) | Helped |
| Train in the stable phase | Large-stepsize GD for nearly homogeneous predictors · General datasets | normalized margin | Helped |
| Use a larger stepsize | GD for two-layer networks · Two-layer networks, stable phase | empirical risk convergence rate: O(1/(tilde η t)) = O(1/(η t)) | Helped |
| Add an initial EoS phase by using a large stepsize | GD for two-layer networks · Linearly separable datasets | empirical risk: O(1/T^2) | Helped |
| Rescale the network with b ≥ 1 rather than b < 1 | Large-stepsize GD for two-layer networks · Linearly separable datasets | empirical risk bound: O(1/T^2) vs. O(b^{-3}/T^2) | Helped |
Surprising findings
- Large stepsize GD can outperform small stepsize GD in optimization speed by an order (1/T^2 versus 1/T) in the separable case.
Limitations
- Sublevel-set condition for entering the stable phase may be empty (e.g., sigmoid networks).
- Acceleration and EoS-phase analysis require linearly separable data and derivative bounded away from zero.
- Margin result only shows improvement, not directional convergence, and is limited to nearly 1-homogeneous predictors.
Experimental setup
Datasets: CIFAR-10 subset (airplane vs automobile, 6,000 samples) XOR dataset (4 samples)
Benchmarks: CIFAR-10 subset (airplane vs automobile) XOR toy dataset
Metrics: empirical logistic risk normalized margin test accuracy
Baselines: small stepsize GD (monotone decrease)
Builds on: Wu et al. 2024 large stepsize GD for logistic regression Lyu and Li 2020 implicit bias of homogeneous networks Ji and Telgarsky 2020 implicit bias of GD Chatterji et al. 2023 edge of stability
Closest papers by meaning
- Gradient Descent Converges Arbitrarily Fast for Logistic Regression via Large and Adaptive Stepsizes (ICML 2025 · similarity 0.90)
- Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks (ICML 2025 · similarity 0.90)
- Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regression (NeurIPS 2025 · similarity 0.89)
- The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents (ICML 2024 · similarity 0.88)
- The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.88)
- Asymptotics of feature learning in two-layer networks after one gradient-step (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.