Research map / Optimization / Implicit bias and generalization
Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with gradient flow on nearly-orthogonal data | Two-layer leaky ReLU network · Nearly-orthogonal classification data | rank: rank at most 2 | Helped |
| Use initialization with standard deviation 50× smaller than Glorot uniform | Two-layer ReLU network trained by SGD · CIFAR-10 | stable rank: 74.0 -> 3.25 | Helped |
| Increase input dimension | Two-layer networks trained by gradient descent · Gaussian mixture model data | stable rank reduction | Helped |
| Decrease initialization scale | Two-layer networks trained by gradient descent · Gaussian mixture model data | stable rank reduction | Helped |
| Use smaller initialization | Two-layer ReLU network trained by SGD · CIFAR-10 | train and test accuracy behavior | No clear effect |
Closest papers by meaning
- Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Data (NeurIPS 2023 · similarity 0.97)
- On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias (NeurIPS 2022 · similarity 0.91)
- Implicit Bias of Mirror Flow for Shallow Neural Networks in Univariate Regression (ICLR 2025 · similarity 0.89)
- Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks (ICML 2025 · similarity 0.89)
- Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks (ICLR 2025 · similarity 0.89)
- Can Implicit Bias Imply Adversarial Robustness? (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.