ICLR 2020 accepted papers on Optimization
52 of the 687 papers accepted at ICLR 2020 (7.6%) are on Optimization, compared with 8.6% at ICLR 2019. They fall into 4 topics.
See ICLR 2020 in the interactive map
Submitting to ICLR? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
Implicit bias and generalization · 20 papers
- Deep Double Descent: Where Bigger Models and More Data Hurt (503 citations)
- Gradient Descent Maximizes the Margin of Homogeneous Neural Networks (102 citations)
- Neural Tangents: Fast and Easy Infinite Neural Networks in Python (92 citations)
- Can gradient clipping mitigate label noise? (69 citations)
- Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks (63 citations)
- Asymptotics of Wide Networks from Feynman Diagrams (51 citations)
- Generalization of Two-layer Neural Networks: An Asymptotic Viewpoint (31 citations)
- Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks (31 citations)
- On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning (30 citations)
- The Break-Even Point on Optimization Trajectories of Deep Neural Networks (28 citations)
- The Implicit Bias of Depth: How Incremental Learning Drives Generalization (22 citations)
- Implicit Bias of Gradient Descent based Adversarial Training on Separable Data (21 citations)
and 8 more in the interactive map.
Nonconvex and smooth optimization · 14 papers
- Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity (83 citations)
- Neural Policy Gradient Methods: Global Optimality and Rates of Convergence (82 citations)
- On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach (51 citations)
- Gradientless Descent: High-Dimensional Zeroth-Order Optimization (37 citations)
- Stochastic AUC Maximization with Deep Neural Networks (36 citations)
- Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets (19 citations)
- Smoothness and Stability in GANs (18 citations)
- Convergence of Gradient Methods on Bilinear Zero-Sum Games (16 citations)
- Reanalysis of Variance Reduced Temporal Difference Learning (16 citations)
- Geometric Analysis of Nonconvex Optimization Landscapes for Overcomplete Learning (14 citations)
- ProxSGD: Training Structured Neural Networks under Regularization and Constraints (10 citations)
- Smooth markets: A basic mechanism for organizing gradient-based learners (6 citations)
and 2 more in the interactive map.
Optimizers · 13 papers
- On the Variance of the Adaptive Learning Rate and Beyond (602 citations)
- Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (162 citations)
- Meta-Learning with Warped Gradient Descent (116 citations)
- Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning (112 citations)
- Accelerating SGD with momentum for over-parameterized learning (46 citations)
- Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform (27 citations)
- Learning to Learn by Zeroth-Order Oracle (9 citations)
- Kernelized Wasserstein Natural Gradient (8 citations)
- Escaping Saddle Points Faster with Stochastic Momentum (5 citations)
- BackPACK: Packing more into Backprop (3 citations)
- Variance Reduction With Sparse Gradients (3 citations)
- Transferring Optimality Across Data Distributions via Homotopy Methods (2 citations)
and 1 more in the interactive map.
Distributed and compressed · 5 papers
- On the Convergence of FedAvg on Non-IID Data (979 citations)
- Don't Use Large Mini-batches, Use Local SGD (212 citations)
- SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum (64 citations)
- Learning Efficient Parameter Server Synchronization Policies for Distributed SGD (4 citations)
- Gap-Aware Mitigation of Gradient Staleness (1 citation)
