Research map / Optimization / Implicit bias and generalization
Embedding Principle of Homogeneous Neural Network for Classification Problem
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that KKT points of the minimum-norm max-margin problem for homogeneous networks map via linear isometries under neuron splitting (FC) and channel splitting (CNN), and that gradient-flow trajectories and omega-limit directions are mapped accordingly.
Problem
Relationship between KKT points of max-margin problems for homogeneous networks of different widths, and between their gradient-flow trajectories, when width is increased by neuron or channel splitting.
Approach
Introduces the KKT point embedding principle: a linear isometry T that preserves network outputs and subgradients (up to positive scaling) maps KKT points of the min-norm max-margin problem of a smaller network to KKT points of a larger network; neuron splitting and a novel channel splitting for CNNs are shown to satisfy this, and gradient flow trajectories are mapped by T.
Key results
- Trajectory error between narrow and wide MLPs trained with gradient descent stays at machine precision (~1e-13) throughout training on a 2D linearly separable dataset.
- The training loss converges toward zero in the same experiment.
- Neuron splitting with coefficients satisfying sum c_i^2 = 1 is proven to be a linear isometry, output-preserving, and subgradient-preserving, hence KKT-point preserving.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Widen an MLP using neuron splitting and initialize it with the mapped narrow-model parameters | MLP trained by gradient descent · 2D linearly separable dataset | trajectory error: ~ 10^{-13} | Helped |
Limitations
- Experiments are limited to small toy settings (2D data, MLPs) and do not test CNNs empirically in the main text.
- Some theorems rely on implicit or unformalized assumptions (e.g., A4 norm divergence, admissibility of chain rules), and the converse direction of the equivalence proof is only sketched.
- Analysis is restricted to positively homogeneous networks without biases.
Experimental setup
Datasets: 2D linearly separable toy dataset
Metrics: trajectory error (L2 norm between embedded and original para training loss
Builds on: KKT conditions for non-smooth problems Clarke subdifferential implicit bias of gradient descent on homogeneous networks Embedding Principle
Closest papers by meaning
- Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks (ICML 2025 · similarity 0.87)
- Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning (NeurIPS 2026 · similarity 0.86)
- Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization (NeurIPS 2024 · similarity 0.86)
- Loss Landscape Characterization of Neural Networks without Over-Parametrization (NeurIPS 2024 · similarity 0.85)
- Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit (NeurIPS 2026 · similarity 0.85)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.