NeurIPS 2020 accepted papers on Optimization
165 of the 1,901 papers accepted at NeurIPS 2020 (8.7%) are on Optimization, compared with 11.1% at NeurIPS 2019. They fall into 4 topics.
See NeurIPS 2020 in the interactive map
Submitting to NeurIPS? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
Nonconvex and smooth optimization · 64 papers
- Natural Policy Gradient Primal-Dual Method for Constrained Markov Decision Processes (64 citations)
- Stochastic Recursive Gradient Descent Ascent for Stochastic Nonconvex-Strongly-Concave Minimax Problems (40 citations)
- Optimal Epoch Stochastic Gradient Descent Ascent Methods for Min-Max Optimization (28 citations)
- Random Reshuffling: Simple Analysis with Vast Improvements (27 citations)
- A Catalyst Framework for Minimax Optimization (22 citations)
- Conformal Symplectic and Relativistic Optimization (19 citations)
- A mean-field analysis of two-player zero-sum games (18 citations)
- A Single-Loop Smoothed Gradient Descent-Ascent Algorithm for Nonconvex-Concave Min-Max Problems (18 citations)
- Stability of Stochastic Gradient Descent on Nonsmooth Convex Losses (18 citations)
- SGD with shuffling: optimal rates without component convexity and large epoch requirements (17 citations)
- Continuous Regularized Wasserstein Barycenters (16 citations)
- Global Convergence and Variance Reduction for a Class of Nonconvex-Nonconcave Minimax Problems (16 citations)
and 52 more in the interactive map.
Optimizers · 45 papers
- AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients (121 citations)
- Estimating Training Data Influence by Tracing Gradient Descent (119 citations)
- Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout (73 citations)
- Why are Adaptive Methods Good for Attention Models? (53 citations)
- Practical Quasi-Newton Methods for Training Deep Neural Networks (46 citations)
- An Improved Analysis of Stochastic Gradient Descent with Momentum (29 citations)
- A Non-Asymptotic Analysis for Stein Variational Gradient Descent (28 citations)
- Projected Stein Variational Gradient Descent (27 citations)
- Convergence of Meta-Learning with Task-Specific Adaptation over Partial Parameters (25 citations)
- Stochastic Optimization for Performative Prediction (25 citations)
- On the distance between two neural networks and the stability of learning (16 citations)
- A Statistical Framework for Low-bitwidth Training of Deep Neural Networks (15 citations)
and 33 more in the interactive map.
Implicit bias and generalization · 42 papers
- The Pitfalls of Simplicity Bias in Neural Networks (98 citations)
- When Do Neural Networks Outperform Kernel Methods? (70 citations)
- Towards Theoretically Understanding Why Sgd Generalizes Better Than Adam in Deep Learning (56 citations)
- On the linearity of large non-linear models: when and why the tangent kernel is constant (46 citations)
- Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification (41 citations)
- On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problems (36 citations)
- Directional convergence and alignment in deep learning (35 citations)
- Bad Global Minima Exist and SGD Can Reach Them (30 citations)
- On the training dynamics of deep networks with $L_2$ regularization (27 citations)
- A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks (24 citations)
- Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate (22 citations)
- Directional Pruning of Deep Neural Networks (21 citations)
and 30 more in the interactive map.
Distributed and compressed · 14 papers
- Minibatch vs Local SGD for Heterogeneous Distributed Learning (42 citations)
- Adaptive Gradient Quantization for Data-Parallel SGD (39 citations)
- A Decentralized Parallel Algorithm for Training Generative Adversarial Nets (38 citations)
- Distributed Training with Heterogeneous Data: Bridging Median- and Mean-Based Algorithms (32 citations)
- Linearly Converging Error Compensated SGD (31 citations)
- Election Coding for Distributed Learning: Protecting SignSGD against Byzantine Attacks (25 citations)
- Distributionally Robust Federated Averaging (18 citations)
- CSER: Communication-efficient SGD with Error Reset (17 citations)
- ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Training (13 citations)
- Optimal and Practical Algorithms for Smooth and Strongly Convex Decentralized Optimization (6 citations)
- Distributed Newton Can Communicate Less and Resist Byzantine Workers (5 citations)
- Debiasing Distributed Second Order Optimization with Surrogate Sketching and Scaled Regularization (2 citations)
and 2 more in the interactive map.
