AI Paper Atlas

Research map / Optimization / Implicit bias and generalization

When Are Solutions Connected in Deep Networks?

Quynh Nguyen, Pierre Bréchet, Marco Mondelli

NeurIPS 2021 · Optimization › Implicit bias and generalization · approach: Neural tangent kernel analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
allow optimizing the weights of the kept subnetwork (Theorem bound, Experiment A) instead of only rescaling after dropping half the neurons (dropout stability, Experiment B) (ablation)FCN trained on CIFAR-10 without dropout · CIFAR-10 FCN, L=6, width 500training loss along connecting pathImproved
use the new connectivity bound (Experiment A) rather than dropout stability (Experiment B) on trained networks (ablation)FCN MNIST (10 hidden layers, width 245), FCN CIFAR-10, VGG-11 CIFAR-10 · MNIST and CIFAR-10training loss upper bound per layerImproved
drop neurons from the first few layers (dropout stability check)SGD-trained networks without dropout · FCN and VGG-11training lossHurt
increase hidden width in a 2-layer FCN, new bound (ablation)FCN with two hidden layers · MNISTmax-over-layers training loss bound: about 200 neurons sufficesImproved
increase hidden width in a 2-layer FCN, new bound (ablation)FCN with two hidden layers · CIFAR-10max-over-layers training loss bound: 500 neurons sufficesImproved
increase width for dropout stability to hold (ablation)2-layer FCN · MNISTdropout-stability loss: 2000 neuronsImproved
increase width for dropout stability to hold (ablation)2-layer FCN · CIFAR-10dropout-stability loss: 3000 neurons still far from stableNo clear effect
increase number of random restarts T_B from 100 to 200 (ablation)dropout stability experiment · minimum loss searchminimum loss: 100 -> 200No clear effect
use dropout during trainingVGG-11 (prior work finding) · dropout stabilitydropout stabilityImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy