Research map / Optimization / Implicit bias and generalization
Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use small initialization and gradient flow with homogeneous activations | Fully connected and convolutional neural networks · Deep homogeneous models | difference between squared norms of adjacent layers | Helped |
| Use positive, diminishing step sizes for unregularized asymmetric matrix factorization | Gradient descent on asymmetric matrix factorization · Rank-r matrix factorization | reconstruction error: ≤ ε | Helped |
| Use constant step size with small random initialization | Gradient descent on asymmetric matrix factorization · Rank-1 matrix factorization | reconstruction error: t = O(log(d/ε)); ||u_t v_tᵀ − M*||F ≤ εσ₁ | Helped |
| Use balanced initialization instead of unbalanced initialization | Gradient descent on a 3-layer ReLU network · 3-layer fully connected network, 1,000 data points | maximum adjacent-layer squared-norm difference: 0.14 vs 9 | Helped |
| Use unbalanced initialization instead of balanced initialization | Gradient descent on a 3-layer ReLU network · 3-layer fully connected network, 1,000 data points | squared-norm ratio between layers | Mixed |
Closest papers by meaning
- Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning (ICLR 2021 · similarity 0.91)
- Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networks (NeurIPS 2022 · similarity 0.90)
- Implicit Regularization in Deep Matrix Factorization (NeurIPS 2019 · similarity 0.90)
- Implicit Regularization for Tubal Tensor Factorizations via Gradient Descent (ICML 2025 · similarity 0.90)
- Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries (ICLR 2025 · similarity 0.89)
- Deep linear networks for regression are implicitly regularized towards flat minima (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.