Research map / Optimization / Implicit bias and generalization

Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning

NeurIPS 2026 · Optimization › Implicit bias and generalization · approach: Shallow ReLU network convergence · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Characterizes max-margin implicit bias of mirror flow in homogeneous neural networks via a new balance equation from convex duality, a Q-margin, and horizon functions, showing sparse (hyperbolic entropy) vs dense (smoothed homogeneous) feature learning.

Problem

Implicit bias of mirror flow (non-Euclidean mirror descent in continuous time) on homogeneous neural networks with exponential loss: which max-margin solution is reached, how fast, and how it shapes features.

Approach

Derives a balance equation for layerwise-separable mirror flow via Fenchel-Young duality, defines a Q-margin using the dual potential Q(∇R) as the mirror analog of the L2 margin, and shows directional convergence to a KKT point of a problem minimizing the squared horizon function.

Key results

• Theory: for alpha in [1,2], loss decays as Theta(1/(t log(t)^{2-alpha/L})) and (alpha Q)^{L/alpha} = Theta(log t), showing slow convergence for non-homogeneous mirror maps.
• Two-layer reformulation: hyperbolic entropy (alpha=1) favors fewer active neurons (L1-type sparsity) while smoothed homogeneous potentials (alpha=p>=2) favor dense activations.
• On CIFAR-10 with VGG-16, hyperbolic entropy achieves the best validation accuracy among the compared mirror maps and yields sparser first-layer weights, with pruning degrading performance less than for the homogeneous potential.

Surprising findings

• Distinct non-homogeneous mirror potentials can converge to the same max-margin solution but at vastly different rates, in some cases exponentially slower in n.
• Convergence to the max-margin direction can be effectively unreachable in finite time for large smoothing parameter lambda relative to dimension.

Limitations

• Max-margin results are asymptotic and directional; finite learning rate, early stopping and generalization are not analyzed.
• Theorem for alpha < 2 (including hyperbolic entropy) not fully extended in the main analysis; hyperbolic case relies on a connection to homogeneous reparameterizations.
• Assumes exponential loss, separability, and layerwise-separable mirror potentials.

Experimental setup

Datasets: CIFAR-10 synthetic student-teacher data

Benchmarks: CIFAR-10

Metrics: validation accuracy L_k-margin loss convergence rate weight magnitude distribution pruning sparsity vs accuracy

Baselines: gradient descent hyperbolic entropy mirror flow smoothened homogeneous mirror potential

Builds on: gradient flow implicit bias (Lyu & Li) mirror flow implicit bias for linear models hyperbolic entropy sparse training Ji & Telgarsky margin maximization

Closest papers by meaning

• Implicit Bias of Mirror Flow for Shallow Neural Networks in Univariate Regression (ICLR 2025 · similarity 0.91)
• Implicit Bias of Mirror Flow on Separable Data (NeurIPS 2024 · similarity 0.88)
• Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks (ICLR 2025 · similarity 0.87)
• Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks (ICML 2025 · similarity 0.86)
• Embedding Principle of Homogeneous Neural Network for Classification Problem (NeurIPS 2025 · similarity 0.86)
• Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.86)