Research map / Optimization / Optimizers
PSMGD: Periodic Stochastic Multi-Gradient Descent for Fast Multi-Objective Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PSMGD, which recomputes MOO dynamic weights (via a QP on stochastic gradients) only every R iterations and reuses them otherwise, achieving standard convergence rates and reduced training time on multi-task learning benchmarks.
Problem
Gradient-manipulation methods for multi-objective optimization are slow because they solve an extra optimization problem for dynamic weights at every iteration.
Approach
Dynamic weights across objectives change little over short intervals, so PSMGD recomputes them periodically (every R steps, with momentum smoothing) and reuses them in between, reducing backpropagation and per-iteration cost while keeping convergence guarantees.
Key results
- On QM-9, PSMGD reaches average test loss <75 in 109.27 minutes-equivalent time, the shortest among compared methods, with Delta m% of 5.55 (vs FairGrad 4.09 and NashMTL 3.82 in the same table column ordering)
- On NYU-v2, PSMGD achieves the best mean-rank-free convergence times, e.g. semantic loss <1.20 in 64.54 min, and ranks third in per-epoch training time (1.85 min)
- On NYU-v2, PSMGD obtains Delta m% of -3.62, behind SDMGrad (-4.84), FairGrad (-4.66), and NashMTL (-4.04) in that table
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PSMGD instead of NashMTL | Multi-task learning on QM-9 · QM-9 test | Mean Rank: 3.82 -> 5.55 | Hurt |
| Use PSMGD instead of FairGrad | Multi-task learning on QM-9 · QM-9 test | Mean per-task performance drop: 57.9 -> 92.4 | Hurt |
| Use PSMGD instead of FAMO | Multi-task learning on QM-9 · QM-9 | Time to average loss < 75: 131.32 -> 109.27 | Helped |
| Use PSMGD instead of FairGrad | Multi-task learning on NYU-v2 · NYU-v2 test | Mean Rank: 3.22 -> 6.11 | Hurt |
| Use PSMGD instead of SDMGrad | Multi-task learning on NYU-v2 · NYU-v2 test | Mean per-task performance drop: -4.84 -> -3.62 | Hurt |
| Use PSMGD instead of LS | Multi-task learning on NYU-v2 · NYU-v2 | Time to semantic loss < 1.40: 32.14 -> 29.28 | Helped |
| Use PSMGD instead of LS | Multi-task learning on NYU-v2 · NYU-v2 | Time to depth loss < 0.65: 46.18 -> 43.07 | Helped |
| Use PSMGD instead of LS | Multi-task learning on NYU-v2 · NYU-v2 | Time to normal loss < 0.20: 30.84 -> 31.11 | Hurt |
Surprising findings
- On NYU-v2 performance metrics, PSMGD is not the best in average Delta m% or mean rank; its advantage is mainly in training speed and convergence time.
Limitations
- Performance on QM-9 and NYU-v2 is comparable rather than uniformly superior; some methods (e.g., FairGrad, SDMGrad) achieve better Delta m% and mean rank
- Convergence guarantees rely on bounded weights, bounded gradients and a bounded-distance assumption; the theory treats the weight-reuse error through a sum of weight changes
Experimental setup
Datasets: QM-9 NYU-v2 Multi-MNIST CelebA CityScapes
Benchmarks: QM-9 NYU-v2
Metrics: Delta m% (average per-task performance drop vs STL) mean rank training time per epoch time to reach loss thresholds
Baselines: STL LS SI DWA UW RLW MGDA PCGrad GradDrop CAGrad IMTL-G NashMTL
Builds on: MGDA stochastic multi-gradient descent CAGrad FAMO
Closest papers by meaning
- MOBO-OSD: Batch Multi-Objective Bayesian Optimization via Orthogonal Search Directions (NeurIPS 2025 · similarity 0.88)
- MGDA Converges under Generalized Smoothness, Provably (ICLR 2025 · similarity 0.87)
- SMOG: Scalable Meta-Learning for Multi-Objective Bayesian Optimization (NeurIPS 2026 · similarity 0.87)
- Improving Pareto Set Learning for Expensive Multi-objective Optimization via Stein Variational Hypernetworks (AAAI 2025 · similarity 0.86)
- MetaBox-v2: A Unified Benchmark Platform for Meta-Black-Box Optimization (NeurIPS 2025 · similarity 0.86)
- ParetoFlow: Guided Flows in Multi-Objective Optimization (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.