Research map / Optimization / Implicit bias and generalization

Implicit Bias of Mirror Flow for Shallow Neural Networks in Univariate Regression

ICLR 2025 · Optimization › Implicit bias and generalization · approach: Implicit bias in linear networks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Characterizes the implicit bias of mirror flow for infinitely wide shallow networks: lazy training with unscaled potentials matches gradient flow, while scaled potentials yield new function-space biases (univariate, absolute value activations).

Problem

Understanding the implicit bias of mirror descent/flow when training wide shallow neural networks for least squares regression, and what solution functions it returns.

Approach

Shows mirror flow with separable unscaled potentials is lazy and matches gradient flow's bias as width grows, giving a function-space variational characterization for univariate ReLU nets; introduces scaled potentials (Phi scaled by n) that remain lazy but outside the kernel regime, producing non-RKHS biases for absolute value networks.

Key results

• Mirror flow with unscaled potentials (x^2, |x|^3+x^2, x^4+x^2) yields the same implicit bias in the infinite-width limit, with L-infinity error to the variational solution decreasing as width grows (4860 neurons in experiments).
• For ReLU networks the implicit bias is a variational problem with curvature penalty weighted by 1/p_B plus boundary terms G2 and G3.
• For scaled potentials with absolute value activations, the bias is the functional integral of D_phi(g''p^{-1}/2, 0) p dx, which is not homogeneous and cannot be an RKHS norm for non-homogeneous phi.

Measured effects reported in the paper

Surprising findings

• Even a constant vertical shift of training labels changes the learned solution substantially, so gradient flow returns natural cubic splines only on specific data.
• Scaled potentials give lazy training yet the kernel matrix does not stay near its initialization, so training is not in the kernel regime.

Limitations

• Results apply only to univariate shallow networks with unit-norm initialized input weights for the function-space characterization.
• Separable potentials only; excludes unnormalized negative entropy and potentials on proper subsets of parameter space.
• Scaled-potential characterization is only proven for absolute value activations, not ReLU.

Experimental setup

Datasets: synthetic univariate regression data

Metrics: L-infinity error between variational solution and trained ne training loss convergence

Baselines: gradient flow natural cubic spline interpolation natural gradient descent

Builds on: gradient flow lazy training analysis Bregman divergence mirror descent hypentropy mirror descent representation cost of ReLU networks

Closest papers by meaning

• Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning (NeurIPS 2026 · similarity 0.91)
• A unifying view on implicit bias in training linear neural networks (ICLR 2021 · similarity 0.91)
• Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias? (ICML 2025 · similarity 0.89)
• Mask in the Mirror: Implicit Sparsification (ICLR 2025 · similarity 0.88)
• Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.88)
• Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks (ICLR 2025 · similarity 0.87)