ICLR 2024 accepted papers on Optimization

Implicit bias and generalization · 24 papers

• Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization (3 citations)
• CausalLM is not optimal for in-context learning (2 citations)
• One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention (2 citations)
• First-order ANIL provably learns representations despite overparametrisation (1 citation)
• Implicit bias of SGD in $L_2$-regularized linear DNNs: One-way jumps from high to low rank (1 citation)
• On the hardness of learning under symmetries (1 citation)
• Stochastic Modified Equations and Dynamics of Dropout Algorithm (1 citation)
• A Precise Characterization of SGD Stability Using Loss Surface Geometry
• Benign Oscillation of Stochastic Gradient Descent with Large Learning Rate
• Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data
• Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression
• How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initialization

and 12 more in the interactive map.

Nonconvex and smooth optimization · 21 papers

• Approximating Nash Equilibria in Normal-Form Games via Stochastic Optimization (2 citations)
• Advancing the Lower Bounds: an Accelerated, Stochastic, Second-order Method with Optimal Adaptation to Inexactness (1 citation)
• Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis (1 citation)
• Decentralized Riemannian Conjugate Gradient Method on the Stiefel Manifold (1 citation)
• Distributionally Robust Optimization with Bias and Variance Reduction (1 citation)
• Nearly $d$-Linear Convergence Bounds for Diffusion Models via Stochastic Localization (1 citation)
• Revisiting the Last-Iterate Convergence of Stochastic Gradient Methods (1 citation)
• Two-timescale Extragradient for Finding Local Minimax Points (1 citation)
• A Primal-Dual Approach to Solving Variational Inequalities with General Constraints
• Constrained Bi-Level Optimization: Proximal Lagrangian Value Function Approach and Hessian-free Algorithm
• Efficient Continual Finite-Sum Minimization
• On Penalty Methods for Nonconvex Bilevel Optimization and First-Order Stochastic Approximation

and 9 more in the interactive map.

Optimizers · 20 papers

• Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (28 citations)
• GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Data (3 citations)
• Synaptic Weight Distributions Depend on the Geometry of Plasticity (3 citations)
• Domain-Inspired Sharpness-Aware Minimization Under Domain Shifts (2 citations)
• Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models (2 citations)
• Accelerating Distributed Stochastic Optimization via Self-Repellent Random Walks (1 citation)
• Magnitude Invariant Parametrizations Improve Hypernetwork Learning (1 citation)
• Stochastic Gradient Descent for Gaussian Processes Done Right (1 citation)
• The Marginal Value of Momentum for Small Learning Rate SGD (1 citation)
• FOSI: Hybrid First and Second Order Optimization
• Ito Diffusion Approximation of Universal Ito Chains for Sampling, Optimization and Boosting
• Lion Secretly Solves a Constrained Optimization: As Lyapunov Predicts

and 8 more in the interactive map.

Distributed and compressed · 7 papers

• Communication-Efficient Gradient Descent-Accent Methods for Distributed Variational Inequalities: Unified Analysis and Local Updates (2 citations)
• EControl: Fast Distributed Optimization with Compression and Error Control (1 citation)
• A Quadratic Synchronization Rule for Distributed Deep Learning
• Det-CGD: Compressed Gradient Descent with Matrix Stepsizes for Non-Convex Optimization
• Flag Aggregator: Scalable Distributed Training under Failures and Augmented Losses using Convex Optimization
• Local Composite Saddle Point Optimization
• On the Effect of Batch Size in Byzantine-Robust Distributed Learning