Research map / Optimization / Implicit bias and generalization
When Are Solutions Connected in Deep Networks?
Quynh Nguyen, Pierre Bréchet, Marco Mondelli
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| allow optimizing the weights of the kept subnetwork (Theorem bound, Experiment A) instead of only rescaling after dropping half the neurons (dropout stability, Experiment B) (ablation) | FCN trained on CIFAR-10 without dropout · CIFAR-10 FCN, L=6, width 500 | training loss along connecting path | Improved |
| use the new connectivity bound (Experiment A) rather than dropout stability (Experiment B) on trained networks (ablation) | FCN MNIST (10 hidden layers, width 245), FCN CIFAR-10, VGG-11 CIFAR-10 · MNIST and CIFAR-10 | training loss upper bound per layer | Improved |
| drop neurons from the first few layers (dropout stability check) | SGD-trained networks without dropout · FCN and VGG-11 | training loss | Hurt |
| increase hidden width in a 2-layer FCN, new bound (ablation) | FCN with two hidden layers · MNIST | max-over-layers training loss bound: about 200 neurons suffices | Improved |
| increase hidden width in a 2-layer FCN, new bound (ablation) | FCN with two hidden layers · CIFAR-10 | max-over-layers training loss bound: 500 neurons suffices | Improved |
| increase width for dropout stability to hold (ablation) | 2-layer FCN · MNIST | dropout-stability loss: 2000 neurons | Improved |
| increase width for dropout stability to hold (ablation) | 2-layer FCN · CIFAR-10 | dropout-stability loss: 3000 neurons still far from stable | No clear effect |
| increase number of random restarts T_B from 100 to 200 (ablation) | dropout stability experiment · minimum loss search | minimum loss: 100 -> 200 | No clear effect |
| use dropout during training | VGG-11 (prior work finding) · dropout stability | dropout stability | Improved |
Closest papers by meaning
- Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis (NeurIPS 2022 · similarity 0.89)
- Input Space Mode Connectivity in Deep Neural Networks (ICLR 2025 · similarity 0.88)
- Understanding Mode Connectivity via Parameter Space Symmetry (ICML 2025 · similarity 0.87)
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.85)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.85)
- On Linear Mode Connectivity of Mixture-of-Experts Architectures (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
