Research map / Optimization / Optimizers

Jacobian Descent for Multi-Objective Optimization

NeurIPS 2026 · Optimization › Optimizers · approach: Randomized gradient estimation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Jacobian descent (JD), a multi-objective generalization of gradient descent defined by an aggregator of the Jacobian, proposes the non-conflicting UPGrad aggregator with convergence-to-Pareto-front guarantees, and applies stochastic sub-Jacobian descent to instance-wise risk minimization on small image classification datasets.

Problem

Gradient descent handles only one objective, while multi-objective and multi-task training with conflicting objectives needs ways to combine per-objective gradients; existing aggregators often lack convergence guarantees under conflict.

Approach

Unify existing gradient-combination methods as aggregators of the Jacobian, and define UPGrad, which projects each row gradient onto the dual cone of all rows and averages the results; this is non-conflicting, linear under scaling, and weighted, yielding convergence to the Pareto front in the smooth convex case.

Key results

• UPGrad gives significant training-loss improvement over the mean aggregator (ERM with SGD) on all six datasets, with the largest gaps on SVHN and CIFAR-10.
• MGDA performs sub-optimally in IWRM, attributed to its sensitivity to small gradients.
• On a convex quadratic test, UPGrad is among the non-conflicting aggregators (with MGDA, DualProj, Nash-MTL) where no objective value increases during optimization.

Surprising findings

• The plain mean aggregator, which directly optimizes the average loss, converges more slowly than several other aggregators in the ERM-objective evaluation.
• Nash-MTL's disappointing results are suspected to stem from instability in the official implementation.

Limitations

• Experiments only on IWRM with moderately conflicting objectives; no benchmark with strongly conflicting tasks.
• Only optimization speed is compared; generalization is not studied.
• Gramian-based computation for general computational graphs is left as an open challenge.

Experimental setup

Datasets: SVHN CIFAR-10 EuroSAT MNIST Fashion-MNIST Kuzushiji-MNIST

Benchmarks: SVHN CIFAR-10 EuroSAT MNIST Fashion-MNIST Kuzushiji-MNIST

Metrics: training loss update similarity to SGD update

Baselines: mean aggregator (SGD/ERM) MGDA PCGrad CAGrad Nash-MTL GradDrop IMTL-G Aligned-MTL RGW DualProj

Builds on: gradient descent MGDA PCGrad CAGrad Nash-MTL

Closest papers by meaning

• On the Convergence of Stochastic Multi-Objective Gradient Manipulation and Beyond (NeurIPS 2022 · similarity 0.87)
• PSMGD: Periodic Stochastic Multi-Gradient Descent for Fast Multi-Objective Optimization (AAAI 2025 · similarity 0.85)
• MGDA Converges under Generalized Smoothness, Provably (ICLR 2025 · similarity 0.84)
• Sarah Frank-Wolfe: Methods for Constrained Optimization with Best Rates and Practical Features (ICML 2024 · similarity 0.84)
• Armijo Line-search Can Make (Stochastic) Gradient Descent Provably Faster (ICML 2025 · similarity 0.84)
• Constrained Bi-Level Optimization: Proximal Lagrangian Value Function Approach and Hessian-free Algorithm (ICLR 2024 · similarity 0.84)