NeurIPS 2021 accepted papers on Optimization
175 of the 2,508 papers accepted at NeurIPS 2021 (7.0%) are on Optimization, compared with 8.7% at NeurIPS 2020. They fall into 4 topics.
See NeurIPS 2021 in the interactive map
Submitting to NeurIPS? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
Nonconvex and smooth optimization · 72 papers
- A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentum (26 citations)
- A Faster Decentralized Algorithm for Nonconvex Minimax Problems (12 citations)
- Efficient Mirror Descent Ascent Methods for Nonsmooth Minimax Problems (12 citations)
- Convergence Rates of Stochastic Gradient Descent under Infinite Noise Variance (10 citations)
- Simple Stochastic and Online Gradient Descent Algorithms for Pairwise Learning (10 citations)
- An Online Method for A Class of Distributionally Robust Optimization with Non-convex Objectives (9 citations)
- Closing the Gap: Tighter Analysis of Alternating Stochastic Gradient Methods for Bilevel Problems (9 citations)
- On Riemannian Optimization over Positive Definite Matrices with the Bures-Wasserstein Geometry (9 citations)
- Provably Faster Algorithms for Bilevel Optimization (9 citations)
- Non-asymptotic convergence bounds for Wasserstein approximation using point clouds (8 citations)
- SUPER-ADAM: Faster and Universal Framework of Adaptive Gradients (8 citations)
- Efficient constrained sampling via the mirror-Langevin algorithm (7 citations)
and 60 more in the interactive map.
Implicit bias and generalization · 48 papers
- Gradient Starvation: A Learning Proclivity in Neural Networks (40 citations)
- Multiple Descent: Design Your Own Generalization Curve (26 citations)
- An Analysis of Constant Step Size SGD in the Non-convex Regime: Asymptotic Normality and Bias (15 citations)
- Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction (13 citations)
- Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms (11 citations)
- Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks (11 citations)
- Early-stopped neural networks are consistent (10 citations)
- Generalization Guarantee of SGD for Pairwise Learning (10 citations)
- Batch Normalization Orthogonalizes Representations in Deep Random Networks (9 citations)
- The Implicit Bias of Minima Stability: A View from Function Space (9 citations)
- Label Noise SGD Provably Prefers Flat Global Minimizers (7 citations)
- Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorization (6 citations)
and 36 more in the interactive map.
Optimizers · 38 papers
- Differentiable Spike: Rethinking Gradient-Descent for Training Spiking Neural Networks (106 citations)
- Conflict-Averse Gradient Descent for Multi-task learning (32 citations)
- Stochastic Solutions for Linear Inverse Problems using the Prior Implicit in a Denoiser (31 citations)
- Differentiable Quality Diversity (29 citations)
- Nonsmooth Implicit Differentiation for Machine-Learning and Optimization (16 citations)
- Efficient Neural Network Training via Forward and Backward Propagation Sparsification (15 citations)
- Practical Large-Scale Linear Programming using Primal-Dual Hybrid Gradient (13 citations)
- Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyond (13 citations)
- On Training Implicit Models (12 citations)
- It Has Potential: Gradient-Driven Denoisers for Convergent Solutions to Inverse Problems (11 citations)
- Profiling Pareto Front With Multi-Objective Stein Variational Gradient Descent (11 citations)
- Stochastic Optimization of Areas Under Precision-Recall Curves with Provable Convergence (9 citations)
and 26 more in the interactive map.
Distributed and compressed · 17 papers
- EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedback (36 citations)
- Asynchronous Decentralized SGD with Quantized and Local Updates (22 citations)
- Rethinking gradient sparsification as total error minimization (18 citations)
- Communication-efficient SGD: From Local SGD to One-Shot Averaging (8 citations)
- Error Compensated Distributed SGD Can Be Accelerated (8 citations)
- An Improved Analysis of Gradient Tracking for Decentralized Machine Learning (7 citations)
- CANITA: Faster Rates for Distributed Convex Optimization with Communication Compression (6 citations)
- A Stochastic Newton Algorithm for Distributed Convex Optimization (4 citations)
- Distributed Zero-Order Optimization under Adversarial Noise (4 citations)
- Preserved central model for faster bidirectional compression in distributed settings (4 citations)
- Distributed Principal Component Analysis with Limited Communication (2 citations)
- ErrorCompensatedX: error compensation for variance reduced algorithms (2 citations)
and 5 more in the interactive map.
