Research map / Learning theory and algorithms / Kernels and regression theory
Deep linear networks can benignly overfit when shallow ones do
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Represent an interpolating linear predictor with a deep linear network rather than the minimum ℓ₂-norm interpolant | Deep linear networks trained with gradient flow · Linear regression, interpolating solutions | conditional variance | No clear effect |
| Use a small first-layer initialization scale (ablation) | Deep linear networks trained to interpolate · Linear regression; simulations | excess risk | Helped |
| Use a small last-layer initialization scale (ablation) | Deep linear networks trained to interpolate · Linear regression; simulations | excess risk | Helped |
| Increase the first-layer initialization scale (ablation) | Deep ReLU networks · ReLU teacher-network simulation | excess risk | Hurt |
| Increase the last-layer initialization scale (ablation) | Deep ReLU networks · ReLU teacher-network simulation | excess risk | No clear effect |
Closest papers by meaning
- Shallow-Deep Networks: Understanding and Mitigating Network Overthinking (ICML 2019 · similarity 0.87)
- On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization (ICML 2018 · similarity 0.84)
- Benign overfitting in leaky ReLU networks with moderate input dimension (NeurIPS 2024 · similarity 0.83)
- On Local Overfitting and Forgetting in Deep Neural Networks (AAAI 2025 · similarity 0.83)
- Provable Tempered Overfitting of Minimal Nets and Typical Nets (NeurIPS 2024 · similarity 0.82)
- Benign Overfitting in Adversarial Training of Neural Networks (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.