Research map / Optimization / Implicit bias and generalization

Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks

ICML 2025 · Optimization › Implicit bias and generalization · approach: Implicit bias in linear networks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Extends implicit bias of gradient descent (normalized margin increase, directional convergence, KKT of margin maximization) from homogeneous to near-homogeneous non-homogeneous deep networks under exponential loss.

Problem

Implicit bias of gradient descent is well understood for homogeneous networks, but unknown for common non-homogeneous networks (residual connections, non-homogeneous activations) under exponential loss.

Approach

Introduces a near-M-homogeneity condition and a strong separability condition, plus a modified margin with a homogeneous-error offset term; shows GF/GD normalized margin increases nearly monotonically, iterates converge in direction, and the limit satisfies KKT conditions of a margin maximization problem on the homogenized network.

Key results

• Under near-M-homogeneity and strong separability, GF risk satisfies L(theta_t) ~ 1/(t (log t)^{2-2/M}) and norm ~ (log t)^{1/M}, with the modified margin an epsilon_t approximation of the normalized margin where epsilon_t = O((log t)^{-1/M}).
• The directional limit of GF and GD exists (finite arc length) and, after rescaling, satisfies the KKT conditions of the margin maximization problem defined by the homogenization of the network.
• A two-layer residual leaky-ReLU network trained by small-stepsize GD from zero initialization is shown to reach the strong separability condition.

Surprising findings

• Strong separability is necessary: a one-sample example shows that without it the risk stays above 1 and the iterates fail to exhibit implicit bias.

Limitations

• Softmax and normalization layers are not covered by the near-homogeneity definition.
• Alignment between parameters and gradients is left as future work; GD results need additional technical conditions (second-derivative bound and strong separability for GD).

Experimental setup

Baselines: Lyu and Li (2020) Ji and Telgarsky (2020) Davis et al. (2020)

Builds on: Lyu and Li (2020) implicit bias of homogeneous networks Lyu and Li normalized margin analysis Davis et al. o-minimal implicit bias Ji and Telgarsky (2020) GD margin improvement

Closest papers by meaning

• Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks (ICLR 2025 · similarity 0.92)
• The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks (ICLR 2023 · similarity 0.91)
• Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization (NeurIPS 2024 · similarity 0.90)
• Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.88)
• Implicit bias of SGD in $L_2$-regularized linear DNNs: One-way jumps from high to low rank (ICLR 2024 · similarity 0.88)
• Non-Singularity of the Gradient Descent Map for Neural Networks with Piecewise Analytic Activations (NeurIPS 2025 · similarity 0.88)