ICML 2021 accepted papers on Optimization
99 of the 1,184 papers accepted at ICML 2021 (8.4%) are on Optimization, compared with 10.9% at ICML 2020. They fall into 4 topics.
See ICML 2021 in the interactive map
Submitting to ICML? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
Nonconvex and smooth optimization · 34 papers
- PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimization (32 citations)
- Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness (16 citations)
- Private Adaptive Gradient Methods for Convex Optimization (14 citations)
- The Limits of Min-Max Optimization Algorithms: Convergence to Spurious Non-Critical Sets (12 citations)
- Optimal Complexity in Decentralized Training (11 citations)
- ADOM: Accelerated Decentralized Optimization Method for Time-Varying Networks (10 citations)
- Learning While Playing in Mean-Field Games: Convergence and Optimality (9 citations)
- Private Stochastic Convex Optimization: Optimal Rates in L1 Geometry (9 citations)
- Accelerated Algorithms for Smooth Convex-Concave Minimax Problems with O(1/k^2) Rate on Squared Gradient Norm (8 citations)
- Stability and Generalization of Stochastic Gradient Methods for Minimax Problems (8 citations)
- A Hybrid Variance-Reduced Method for Decentralized Stochastic Non-Convex Optimization (7 citations)
- Learner-Private Convex Optimization (6 citations)
and 22 more in the interactive map.
Optimizers · 30 papers
- Bilevel Optimization: Convergence Analysis and Enhanced Design (31 citations)
- Multiplicative Noise and Heavy Tails in Stochastic Optimization (30 citations)
- How Do Adam and Training Strategies Help BNNs Optimization (24 citations)
- Optimizing persistent homology based functions (15 citations)
- A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance (12 citations)
- A Value-Function-based Interior-point Method for Non-convex Bi-level Optimization (12 citations)
- GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training (11 citations)
- Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization (9 citations)
- When All We Need is a Piece of the Pie: A Generic Framework for Optimizing Two-way Partial AUC (8 citations)
- Efficient Differentiable Simulation of Articulated Bodies (6 citations)
- Guarantees for Tuning the Step Size using a Learning-to-Learn Approach (6 citations)
- A Gradient Based Strategy for Hamiltonian Monte Carlo Hyperparameter Optimization (5 citations)
and 18 more in the interactive map.
Implicit bias and generalization · 23 papers
- The Heavy-Tail Phenomenon in SGD (34 citations)
- How Does Loss Function Affect Generalization Performance of Deep Learning? Application to Human Age Estimation (14 citations)
- Tensor Programs IIb: Architectural Universality Of Neural Tangent Kernel Training Dynamics (13 citations)
- Toward Better Generalization Bounds with Locally Elastic Stability (12 citations)
- Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization (11 citations)
- How to Learn when Data Reacts to Your Model: Performative Gradient Descent (11 citations)
- On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widths (9 citations)
- Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent (6 citations)
- On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent (6 citations)
- Scaling Properties of Deep Residual Networks (6 citations)
- Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programs (5 citations)
- Quantifying the Benefit of Using Differentiable Learning over Tangent Kernels (4 citations)
and 11 more in the interactive map.
Distributed and compressed · 12 papers
- Quasi-global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data (32 citations)
- Distributed Second Order Methods with Fast Rates and Compressed Communication (19 citations)
- 1-bit Adam: Communication Efficient Large-Scale Training with Adam’s Convergence Speed (14 citations)
- MARINA: Faster Non-Convex Distributed Learning with Compression (14 citations)
- Stochastic Sign Descent Methods: New Algorithms and Better Theory (10 citations)
- BASGD: Buffered Asynchronous SGD for Byzantine Learning (6 citations)
- Accelerating Gossip SGD with Periodic Global Averaging (3 citations)
- Communication-Efficient Distributed Optimization with Quantized Preconditioners (2 citations)
- Newton Method over Networks is Fast up to the Statistical Precision
- Asynchronous Decentralized Optimization With Implicit Stochastic Variance Reduction
- Asynchronous Distributed Learning : Adapting to Gradient Delays without Prior Knowledge
- Byzantine-Resilient High-Dimensional SGD with Local Iterations on Heterogeneous Data
