ICLR 2021 accepted papers on Optimization
52 of the 860 papers accepted at ICLR 2021 (6.0%) are on Optimization, compared with 7.6% at ICLR 2020. They fall into 4 topics.
See ICLR 2021 in the interactive map
Submitting to ICLR? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
Implicit bias and generalization · 24 papers
- On the Origin of Implicit Regularization in Stochastic Gradient Descent (39 citations)
- A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima (38 citations)
- A unifying view on implicit bias in training linear neural networks (27 citations)
- Entropic gradient descent algorithms and wide flat minima (26 citations)
- How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks? (23 citations)
- Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (21 citations)
- Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning (19 citations)
- On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers (16 citations)
- Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime (16 citations)
- Noise against noise: stochastic label noise helps combat inherent label noise (15 citations)
- The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers (15 citations)
- Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics (14 citations)
and 12 more in the interactive map.
Optimizers · 15 papers
- AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights (93 citations)
- Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models (73 citations)
- Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients (11 citations)
- Randomized Automatic Differentiation (10 citations)
- RMSprop converges with proper hyper-parameter (9 citations)
- Convex Potential Flows: Universal Probability Distributions with Optimal Transport and Convex Optimization (7 citations)
- Meta-learning with negative learning rates (4 citations)
- Taming GANs with Lookahead-Minmax (4 citations)
- NOVAS: Non-convex Optimization via Adaptive Stochastic Search for End-to-end Learning and Control (3 citations)
- Understanding the effects of data parallelism and sparsity on neural network training (3 citations)
- Accelerating Convergence of Replica Exchange Stochastic Gradient MCMC via Variance Reduction (2 citations)
- DDPNOpt: Differential Dynamic Programming Neural Optimizer (2 citations)
and 3 more in the interactive map.
Nonconvex and smooth optimization · 8 papers
- Linear Last-iterate Convergence in Constrained Saddle-point Optimization (24 citations)
- On the Impossibility of Global Convergence in Multi-Loss Optimization (13 citations)
- Sharper Generalization Bounds for Learning with Gradient-dominated Objective Functions (10 citations)
- Adaptive Extra-Gradient Methods for Min-Max Optimization and Games (8 citations)
- Proximal Gradient Descent-Ascent: Variable Convergence under KŁ Geometry (8 citations)
- Local Convergence Analysis of Gradient Descent Ascent with Finite Timescale Separation (7 citations)
- The Role of Momentum Parameters in the Optimal Convergence of Adaptive Polyak's Heavy-ball Methods (6 citations)
- Fast convergence of stochastic subgradient method under interpolation
Distributed and compressed · 5 papers
- Distributed Momentum for Byzantine-resilient Stochastic Gradient Descent (21 citations)
- Linear Convergent Decentralized Optimization with Compression (21 citations)
- A Better Alternative to Error Feedback for Communication-Efficient Distributed Learning (18 citations)
- Byzantine-Resilient Non-Convex Stochastic Gradient Descent (17 citations)
- Multi-Level Local SGD: Distributed SGD for Heterogeneous Hierarchical Networks (10 citations)
