Research map / Optimization / Implicit bias and generalization
Exploring The Loss Landscape Of Regularized Neural Networks Via Convex Duality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses the convex reformulation and its dual of regularized ReLU networks to characterize optimal sets, stationary points, the phase-transitional connectivity of optimal solutions, and non-unique minimum-norm interpolators, with extensions to vector-valued and parallel three-layer networks.
Problem
Understanding the loss landscape and global optimal set of regularized (weight-decay) neural networks with ReLU activation, including connectivity of optimal solutions and uniqueness of minimum-norm interpolators.
Approach
Characterizes the optimal polytope via the dual optimum of the cone-constrained group LASSO convex reformulation, derives critical widths giving a staircase of connectivity as width changes, and constructs non-unique interpolator datasets motivated by the dual problem; also shows free skip connections, bias penalization, and d=1 are all needed for uniqueness.
Key results
- Gradient descent on the toy problems finds multiple distinct optimal interpolators with equal cost, supporting the continuum-of-optima claim.
- Phase transition in connectivity of the optimal set is established at critical widths m* and M* with connectedness for m >= min{m*+M*, n+1}.
- Non-uniqueness of minimum-norm interpolators is shown for 2D inputs (SB and SNB settings) and for d=1 without skip connections (NSB).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase width from m = m* to m ≥ m*+1 | Two-layer scalar-output ReLU network · Optimal parameter set | connectivity | Helped |
| increase width from m = M* to m ≥ M*+1 | Two-layer scalar-output ReLU network · Optimal parameter set | connectivity | Helped |
| use width m ≥ min{m*+M*, n+1} | Two-layer scalar-output ReLU network · Optimal parameter set | connectivity | Helped |
| use width m ≥ n+1 | Two-layer scalar-output ReLU network · Loss landscape | path to any global optimum with nonincreasing loss | Helped |
| add a free skip connection, regularize the bias, and use unidimensional data | Minimum-norm interpolator · Uniqueness of optimal interpolator | uniqueness | Helped |
| use data with dimension greater than one | Minimum-norm interpolator with free skip connection and regularized bias · Minimum-norm interpolation | uniqueness | Hurt |
| remove free skip connections while retaining regularized bias | Minimum-norm interpolator · Unidimensional data, d = 1 | uniqueness | Hurt |
| use width m ≥ nc + 1 | Two-layer vector-valued neural network · Loss landscape | path to any global optimum with nonincreasing loss | Helped |
| increase width m | Toy two-layer scalar-output network · Toy regularized regression, two data points | connectivity of optimal solutions | Helped |
Surprising findings
- Free skip connections can change the qualitative uniqueness behavior of minimum-norm interpolators, contrary to prior empirical belief that they do not matter.
- Uniqueness of minimum-norm interpolators does not hold in dimensions greater than one even with free skip connections.
Limitations
- Results are for ReLU and scalar-output two-layer networks in the main text; extensions to deeper architectures are limited to parallel networks.
- Uniqueness of minimum-norm interpolators in high dimensions remains open.
- Experiments are small toy problems run on a laptop.
Experimental setup
Datasets: synthetic toy regression data (two points) synthetic 5-point interpolation dataset
Metrics: objective value connectivity of optimal solution sets
Builds on: convex reformulation of two-layer ReLU networks (Pilanci & Ergen 2020) optimal polytope characterization (Ergen & Pilanci-style convex analysis) mode connectivity
Closest papers by meaning
- The Hidden Convex Optimization Landscape of Regularized Two-Layer ReLU Networks: an Exact Characterization of Optimal Solutions (ICLR 2022 · similarity 0.92)
- Visualizing the Loss Landscape of Neural Nets (NeurIPS 2018 · similarity 0.89)
- Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks (ICML 2020 · similarity 0.89)
- SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures (NeurIPS 2025 · similarity 0.88)
- Flat Channels to Infinity in Neural Loss Landscapes (NeurIPS 2025 · similarity 0.88)
- Loss Landscape Characterization of Neural Networks without Over-Parametrization (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.