Research map / Optimization / Optimizers

Delve into the Applicability of Advanced Optimizers for Multi-Task Learning

NeurIPS 2026 · Optimization › Optimizers · approach: Multi-objective gradient methods · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes APT, which adds a curvature-driven adaptive momentum coefficient (based on cosine similarity of consecutive MTL gradients) and a light direction preservation aggregation to make advanced optimizers (Adam, Muon) compatible with optimization-based multi-task learning.

Problem

Momentum (EMA) in advanced optimizers such as Adam and Muon dilutes the de-conflicting effect of optimization-based multi-task learning methods, and Muon's orthogonalization requires a high-rank, well-balanced input gradient.

Approach

Set the momentum coefficient per step from the cosine similarity of consecutive MTL gradients (high momentum when curvature is low, low momentum when high), and aggregate task gradients with sigmoid-gated cosine weights to preserve directional diversity as input to Muon's orthogonalization.

Key results

• On NYUv2, APT-Muon reaches 42.47 segmentation mIoU and Δm% of -5.54, the best among compared methods on that benchmark.
• On Cityscapes, APT-Muon achieves MR 1.00 and Δm% -3.09, versus the best prior baseline COST at MR 4.25 and Δm% 4.30.
• APT-FAMO reports the best QM9 Δm% at 55.5 (with large variance ±7.79), and APT-Fair reports the best CelebA MR at 6.75.

Surprising findings

• Momentum (EMA) substantially reduces the de-conflicting effect of gradient-based MTL methods, with lower β1 improving results across most methods.
• Muon performs poorly with MGDA and FairGrad, but well with CAGrad and Nash-MTL, which the authors attribute to effective rank differences.

Limitations

• On CelebA, pairing APT with FairGrad slightly degrades performance, which the authors attribute to the complexity of 40 tasks.
• Some APT variants show large standard deviations (e.g., APT-FAMO on QM9 ±7.79), and the gains are not uniform across baselines.
• Theoretical results rely on standard assumptions and bound tracking error in asymptotic averages rather than giving concrete rate guarantees.

Experimental setup

Datasets: Cityscapes NYUv2 CelebA QM9

Benchmarks: Cityscapes NYUv2 CelebA QM9

Metrics: mean rank (MR) Δm% mIoU pixel accuracy depth error surface normal angle distance MAE

Baselines: LS SI RLW DWA UW MGDA GradDrop PCGrad CAGrad IMTL Nash-MTL FAMO

Builds on: MGDA CAGrad FAMO FairGrad Muon Adam

Closest papers by meaning

• Towards Consistent Multi-Task Learning: Unlocking the Potential of Task-Specific Parameters (CVPR 2025 · similarity 0.91)
• Continual Optimization with Symmetry Teleportation for Multi-Task Learning (NeurIPS 2025 · similarity 0.89)
• Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learning (ICML 2024 · similarity 0.88)
• MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization (NeurIPS 2025 · similarity 0.88)
• Fair Resource Allocation in Multi-Task Learning (ICML 2024 · similarity 0.87)
• MTLoRA: Low-Rank Adaptation Approach for Efficient Multi-Task Learning (CVPR 2024 · similarity 0.87)