Research map / Optimization / Implicit bias and generalization

Implicit Bias of Mirror Flow on Separable Data

NeurIPS 2024 · Optimization › Implicit bias and generalization · approach: Implicit bias in linear networks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Characterises the implicit bias of continuous-time mirror descent (mirror flow) on linearly separable classification with exponential-tailed losses: iterates converge in direction to a max-margin classifier under a norm given by the potential's horizon function.

Problem

Which separating solution mirror flow (continuous-time mirror descent) converges to on linearly separable classification data, when the potential is not L-homogeneous and its geometry changes as iterates diverge.

Approach

Defines a horizon function phi_infty as the Minkowski gauge of the limiting normalised sublevel set of the potential; the iterates' directions converge to the phi_infty-max-margin classifier. For separable potentials phi_infty(b)=lambda lim_{eta->0} eta*varphi^{-1}(phi(b/eta)) gives an explicit formula.

Key results

• Mirror flow with the l2, cosh-entropy and hyperbolic-entropy potentials converges in direction to the l2-, l-infinity- and l1-max-margin classifiers respectively on a toy 2d dataset, matching the theory.
• For separable potentials, phi_infty(b)=lambda lim_{eta->0} eta*varphi^{-1}(phi(b/eta)) gives explicit horizon functions; phi_infty is proportional to the l_p norm for L-homogeneous potentials ||.||_p^p.

Measured effects reported in the paper

Surprising findings

• Potentials with non-coercive gradients (e.g. diagonal linear networks of depth D>2) have horizon functions proportional to l1 regardless of depth, yet predictors converge to KKT points of l_{2/D}-max-margin, so the coercive gradient assumption is necessary.

Limitations

• Results are purely asymptotic; convergence rates to the max-margin direction are not characterised.
• Does not cover potentials defined on a strict subset of R^d (e.g. log-barrier, negative entropy) or with non-coercive gradients.
• Uniqueness of the phi_infty-max-margin solution requires assumptions when the horizon unit ball is not strictly convex; the discrete mirror descent case is not analysed.

Experimental setup

Datasets: synthetic 2d separable dataset (40 positive, 40 negative points)

Metrics: training loss directional convergence of normalised iterates limit direction vs phi_infty-max-margin

Baselines: gradient flow (l2 potential)

Builds on: mirror descent gradient descent implicit bias on separable data diagonal linear networks

Closest papers by meaning

• Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning (NeurIPS 2026 · similarity 0.88)
• Mirror Descent with Relative Smoothness in Measure Spaces, with application to Sinkhorn and EM (NeurIPS 2022 · similarity 0.88)
• Mirror Descent Maximizes Generalized Margin and Can Be Implemented Efficiently (NeurIPS 2022 · similarity 0.88)
• Multiclass Loss Geometry Matters for Generalization of Gradient Descent in Separable Classification (NeurIPS 2025 · similarity 0.85)
• Implicit Bias of Mirror Flow for Shallow Neural Networks in Univariate Regression (ICLR 2025 · similarity 0.85)
• Mirror Descent-Ascent for mean-field min-max problems (NeurIPS 2026 · similarity 0.84)