Research map / Optimization / Optimizers
Multi-objective gradient methods
Provides stochastic gradient algorithms with convergence guarantees for multi-objective and multi-task learning.
28 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2019–2026) · an approach in Optimizers, keyword cluster “differentiable · functions · loss”
State History Tried Baselines Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
8 papers in 2023–24 and 11 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 3.3% to 3.8%.
Open questions the data points to
- Papers report MGDA on NYU-v2 segmentation (mIoU) anywhere from 30.47 to 46.39: comparisons against it depend on whose number you take.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2019 · AutoLoss: Learning Discrete Schedule for Alternate Optimization (ICLR 2019 · 29 citations)
- 2020 · Multi-Task Learning with User Preferences: Gradient Descent with Controlled Ascent in Pareto Optimization (ICML 2020 · 37 citations)
- 2021 · Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models (ICLR 2021 · 73 citations)
- 2022 · GAL: Gradient Assisted Learning for Decentralized Multi-Organization Collaborations (NeurIPS 2022 · 6 citations)
- 2023 · FAMO: Fast Adaptive Multitask Optimization (NeurIPS 2023 · 13 citations)
- 2024 · Runtime Analysis of the SMS-EMOA for Many-Objective Optimization (AAAI 2024 · 38 citations)
- 2025 · PSMGD: Periodic Stochastic Multi-Gradient Descent for Fast Multi-Objective Optimization (AAAI 2025 · 6 citations)
- 2026 · Delve into the Applicability of Advanced Optimizers for Multi-Task Learning (NeurIPS 2026 · 0 citations)
Earliest papers here
- AutoLoss: Learning Discrete Schedule for Alternate Optimization (ICLR 2019 · 29 citations)
- Multi-Task Learning with User Preferences: Gradient Descent with Controlled Ascent in Pareto Optimization (ICML 2020 · 37 citations)
- Growing Adaptive Multi-hyperplane Machines (ICML 2020 · 0 citations)
What has been tried
21 papers here report 214 measured changes (a component added, removed or swapped, compared with a baseline): helped 64%, no clear effect 8%, hurt 26%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add performance-informed weighting to LS | NYUv2 · ∆m(%) | 5.59 -> -0.06 | Helped | Revisiting Fairness in Multitask Learning: A Performance-Driven Approach for Variance Reduction (CVPR 2025 · 2 citations) |
| Add Δ optimization | 8-task vision benchmark · average accuracy | 71.9 -> 80.7 | Helped | Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · 0 citations) |
| Add LSTM history summarization | Partial-obobservable CartPole · return | Helped | Aligned Multi Objective Optimization (ICML 2025 · 0 citations) | |
| Use UAS rather than USA task order | CoVoST-2 Conformer and Whisper-medium speech recognition · WER | Helped | Objective Soups: Multilingual Multi-Task Modeling for Speech Processing (NeurIPS 2025 · 0 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| MGDA | NYU-v2 segmentation | mIoU | 30.47 – 46.39 | 2 papers, e.g. NeurIPS 2021 and NeurIPS 2023 |
What reviewers will expect
Datasets and baselines named in the 9 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- QM9 · 3 of 9 papers
- NYUv2 · 3 of 9 papers
- CelebA · 3 of 9 papers
- CityScapes · 3 of 9 papers
Baselines
- LS · 3 of 9 papers
- SI · 3 of 9 papers
- DWA · 3 of 9 papers
- UW · 3 of 9 papers
- RLW · 3 of 9 papers
- MGDA · 3 of 9 papers
- PCGrad · 3 of 9 papers
- GradDrop · 3 of 9 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 19 papers; from OpenAlex and the papers' affiliations).
- Rensselaer Polytechnic Institute · 3 papers
- IBM · 2 papers
Use it
Download BibTeX of 26 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Differentiating through embedded optimization (117 papers)
- Surrogate losses and robust solvers (119 papers)
- Constrained and non-backprop training (97 papers)
- Randomized gradient estimation (77 papers)
Elsewhere in the topic
- Stochastic gradient MCMC (142 papers)
- Memory-efficient LLM optimizers (79 papers)
- Adam convergence and adaptive learning rates (78 papers)
- Natural gradient and Fisher approximations (77 papers)
- SGD momentum and step-size schedules (72 papers)
- Second-order and structured preconditioning (70 papers)
- Stein variational gradient descent (47 papers)
- Gradient-based meta-learning theory (40 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).