Research map / Optimization / Optimizers
Surrogate losses and robust solvers
Proposes tailored losses, quantization regularizers, and fast solvers for specific learning problems.
119 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Optimizers, keyword cluster “differentiable · functions · loss”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
31 papers in 2023–24 and 29 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 12.8% to 10.0%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Fits Like a Glove: Rapid and Reliable Hand Shape Personalization (CVPR 2016 · 86 citations)
- 2017 · Multiscale Quantization for Fast Similarity Search (NeurIPS 2017 · 46 citations)
- 2018 · Mean-Variance Loss for Deep Age Estimation From a Face (CVPR 2018 · 242 citations)
- 2019 · Deep Metric Learning to Rank (CVPR 2019 · 255 citations)
- 2020 · Transparent Classification with Multilayer Logical Perceptrons and Random Binarization (AAAI 2020 · 33 citations)
- 2021 · Stochastic Solutions for Linear Inverse Problems using the Prior Implicit in a Denoiser (NeurIPS 2021 · 31 citations)
- 2022 · HerosNet: Hyperspectral Explicable Reconstruction and Optimal Sampling Deep Network for Snapshot Compressive Imaging (CVPR 2022 · 73 citations)
- 2023 · FedMDFG: Federated Learning with Multi-Gradient Descent and Fair Guidance (AAAI 2023 · 27 citations)
- 2024 · LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP (CVPR 2024 · 30 citations)
- 2025 · Balanced Rate-Distortion Optimization in Learned Image Compression (CVPR 2025 · 7 citations)
- 2026 · Row-Stochastic Matrices Can Provably Outperform Doubly Stochastic Matrices in Decentralized Learning (ICML 2026 · 0 citations)
Earliest papers here
- Fits Like a Glove: Rapid and Reliable Hand Shape Personalization (CVPR 2016 · 86 citations)
- Learning from Multiway Data: Simple and Efficient Tensor Regression (ICML 2016 · 27 citations)
- Fast Lasso Algorithm via Selective Coordinate Descent (AAAI 2016 · 22 citations)
What has been tried
101 papers here report 895 measured changes (a component added, removed or swapped, compared with a baseline): helped 69%, no clear effect 8%, hurt 18%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add text knowledge by learning class-wise blending parameters instead of setting αk = 0 | 11 benchmarks, 1-shot · accuracy | 35.55 -> 63.43 | Helped | LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP (CVPR 2024 · 30 citations) |
| increase the inverse Hessian approximation size | AAPM, 32 views, no noise · PSNR (dB) | 35.69 -> 38.11 -> 39.37 -> 39.51 | Helped | QN-Mixer: A Quasi-Newton MLP-Mixer Model for Sparse-View CT Reconstruction (CVPR 2024 · 11 citations) |
| Add R-on (implicit online Hessian approximation) to joint GA+FT | CIFAR-10 random subset forgetting (10%), ResNet-18 · Avg.D | 3.20 -> 2.00 | Helped | Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement (NeurIPS 2024 · 8 citations) |
| Remove weight renormalization (ct scaling) from the proposed solutions | Kodak, lambda=0.013 · R-D curve convergence | Helped | Balanced Rate-Distortion Optimization in Learned Image Compression (CVPR 2025 · 7 citations) | |
| reduce batch size from 32 to 1 | MNIST, training from scratch, σ=0.03 · test accuracy | 0.87 to 0.64 | Hurt | Certified Machine Unlearning via Noisy Stochastic Gradient Descent (NeurIPS 2024 · 5 citations) |
| add the improved noise schedule | CIFAR-10, N=10 · FID | 16.86 -> 13.25 | Helped | Interpreting and Improving Diffusion Models from an Optimization Perspective (ICML 2024 · 4 citations) |
| Use softsign rather than entmoid or sigmoid split activation | 19 binary classification datasets · Normalized Mean macro F1 | 0.2207 -> 0.7906 | Helped | GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Data (ICLR 2024 · 3 citations) |
| Set moving-average β to 0.9 rather than 0.99 | BBBP · ROC-AUC | 0.961 ± 0.012 -> 0.958 ± 0.014 | Hurt | Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models (ICLR 2024 · 2 citations) |
| use a proximal step on data fidelity | CBSD68 inpainting, random mask p=0.5 · PSNR | 31.65 -> 31.94 | Helped | Plug-and-Play image restoration with Stochastic deNOising REgularization (ICML 2024 · 1 citation) |
| Increase magnitude regularization strength from λ=0 to λ=1 | MNIST · Generalization gap | 0.0736 -> 0.0464 | Helped | Approximating Metric Magnitude of Point Sets (AAAI 2025 · 0 citations) |
| Vary the noise assumed when estimating privacy-loss rankings | SVHN, ResNet-18 · Spearman correlation | 0.70 (p = 0.0); 0.99 (p = 0.0); 0.99 (p = 0.0) | No clear effect | Leveraging Per-Instance Privacy for Machine Unlearning (ICML 2025 · 0 citations) |
| Add sparse deviations | Super-resolution, CelebA · LPIPS, PSNR, and SSIM | Helped | Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models (ICML 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 16 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 7 of 16 papers
- ImageNet · 4 of 16 papers
- CIFAR100 · 2 of 16 papers
Baselines
- L1-Sparse · 2 of 16 papers
- SalUn · 2 of 16 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 60 papers; from OpenAlex and the papers' affiliations).
- Google · 4 papers
- Massachusetts Institute of Technology · 3 papers
- Purdue University · 3 papers
- The Chinese University of Hong Kong · 2 papers
- EPFL · 2 papers
- National University of Singapore · 2 papers
- Southern University of Science and Technology · 2 papers
- Jilin University · 2 papers
- Texas A&M University - College Station · 2 papers
- University of Toronto · 2 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Differentiating through embedded optimization (117 papers)
- Constrained and non-backprop training (97 papers)
- Randomized gradient estimation (77 papers)
- Multi-objective gradient methods (28 papers)
Elsewhere in the topic
- Stochastic gradient MCMC (142 papers)
- Memory-efficient LLM optimizers (79 papers)
- Adam convergence and adaptive learning rates (78 papers)
- Natural gradient and Fisher approximations (77 papers)
- SGD momentum and step-size schedules (72 papers)
- Second-order and structured preconditioning (70 papers)
- Stein variational gradient descent (47 papers)
- Gradient-based meta-learning theory (40 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).