Research map / Optimization / Implicit bias and generalization
Explicit and Implicit Graduated Optimization in Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Experimentally evaluates explicit graduated optimization with optimal polynomial noise decay on 16 benchmark functions and DNNs, proves Rastrigin's function is a new sigma-nice function, and extends implicit graduated optimization (noise from SGD) to SGD with momentum (SHB/NSHB) with convergence analysis and CIFAR100/ImageNet experiments.
Problem
Graduated optimization for nonconvex training objectives: whether the theoretically optimal noise decay works in practice for explicit smoothing on benchmark functions and DNNs, and whether implicit graduated optimization via SGD noise extends to SGD with momentum.
Approach
Proves Rastrigin's function is a new sigma-nice function, empirically tests explicit graduated optimization with optimal polynomial noise decay on benchmarks and DNNs, and extends implicit graduated optimization to SHB and NSHB by tuning learning rate, batch size and momentum so that the stochastic-noise smoothing level decays at the optimal polynomial rate.
Key results
- On 16 benchmark functions at D=50, explicit graduated optimization with optimal decay (EGO nice) reached the lowest value on several functions (e.g., Rastrigin's 2.26E-02 vs GA 2.44E+01 and PSO 1.51E+02).
- Explicit graduated optimization with noise on ResNet18/CIFAR100 did not yield lower loss than vanilla SGD, so the authors saw no benefit for DNNs.
- Implicit graduated optimization with SHB outperformed vanilla SHB in test accuracy and training loss on ResNet18/CIFAR100, and similar results held for WideResNet-28-10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EGO(nice) instead of EGO(geo) (ablation) | Explicit graduated optimization · Traditional global optimization functions, D=50 | optimum objective value: 1.80E+01 -> 4.04E-03 (Ackley's); 1.85E+00 -> 2.26E-02 (Rastrigin's) | Helped |
| use EGO(nice) instead of the strongest listed comparator | Explicit graduated optimization · Traditional global optimization functions, D=50 | optimum objective value: 4.04E-03 vs 1.80E+01 (Ackley's); 1.25E-01 vs 2.72E-04 (Alpine1); 2.26E-02 vs 1.85E+00 (Rastrigin's) | Mixed |
| add explicit graduated optimization noise (initial δ₁ of 0.1, 0.01, or 0.001) | ResNet18 trained with SGD · CIFAR100 | training loss | Hurt |
| decrease learning rate, increase batch size, or use the hybrid schedule (ablation) | SGD · ResNet18 on CIFAR100 | test accuracy | Helped |
| increase batch size as part of the implicit graduated optimization schedule (ablation) | SGD · ResNet18 on CIFAR100 | test accuracy | Helped |
| reduce smoothing by updating hyperparameters | SHB · ResNet18 on CIFAR100 | test accuracy | Helped |
| reduce smoothing by updating hyperparameters | SHB · ResNet18 on CIFAR100 | training loss | Helped |
| use polynomial learning-rate decay with p ∈ (0,1] (ablation) | SHB and NSHB · CIFAR100 and ImageNet | training loss | Helped |
| use a polynomial decay scheduler with power less than or equal to 1 (ablation) | SHB · ResNet34 on ImageNet | training loss | Helped |
| use polynomial decay with p=0.1 rather than p ∈ {0.5, 0.9, 1} (ablation) | SHB and NSHB · CIFAR100 | training loss | Hurt |
Surprising findings
- Explicit graduated optimization with optimal noise decay did not outperform vanilla SGD on ResNet18 training, and the authors could not explain the cause.
- On the benchmark table, EGO with the geometric (suboptimal) decay sometimes performed worse than optimal decay, yet PSO or GA beat EGO on many functions.
Limitations
- Explicit graduated optimization was not effective for DNN training, with the cause unknown and possibly tied to high dimensionality.
- Convergence guarantees rely on new sigma-nice function assumptions and hold only in expectation with O(1/epsilon^(1/p)) rounds.
- Experiments are limited to ResNet18, WideResNet-28-10 and ResNet34 on CIFAR100 and ImageNet with few runs (three).
Experimental setup
Datasets: CIFAR100 ImageNet
Benchmarks: CIFAR100 ImageNet Ackley's Alpine1 Drop-Wave Ellipsoid Griewank HappyCat HGBat Modified Ridge Rastrigin's Rosenbrock's
Metrics: average optimum value over 50 runs training loss test accuracy
Baselines: genetic algorithm particle swarm optimization vanilla SGD vanilla SHB geometric noise decay
Builds on: graduated optimization implicit graduated optimization with SGD stochastic heavy ball normalized stochastic heavy ball
Closest papers by meaning
- A General Regret Bound of Preconditioned Gradient Method for DNN Training (CVPR 2023 · similarity 0.85)
- GradMax: Growing Neural Networks using Gradient Information (ICLR 2022 · similarity 0.85)
- Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise (ICLR 2025 · similarity 0.85)
- Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search (ICML 2019 · similarity 0.84)
- Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks (NeurIPS 2018 · similarity 0.84)
- Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks (ICLR 2018 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.