Research map / Optimization / Optimizers
Towards Consistent Multi-Task Learning: Unlocking the Potential of Task-Specific Parameters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConsMTL, a bi-level multi-task learning method that uses task-specific parameters, guided by an extra alignment loss, to make their gradients on shared parameters consistent with a joint gradient-aggregation update, reducing gradient conflicts.
Problem
Gradient conflicts across tasks in multi-task learning; prior gradient-based and loss-based methods optimize only shared parameters and ignore how task-specific parameters affect shared-parameter gradients.
Approach
Model MTL as bi-level optimization: an upper level finds a fair joint update direction for shared parameters via a closed-form-like inverse-utility objective solved with Lagrange multipliers; a lower level adds an extra loss -λ<∇z ℓi, dz> to each task-specific parameter set so their shared-representation gradients align with the joint direction.
Key results
- On NYUv2 (3-task), ConsMTL achieves mean rank 2.78 and best Δm% of -6.72%, the only method beating STL on all 3 tasks and 9 metrics.
- On CelebA (40-task), ConsMTL reaches Δm% of -1.42% (only method below STL) with mean rank 3.40.
- On QM9 (11-task), ConsMTL attains Δm% of 2.00 vs. 4.55 for the next best (Nash-MTL) on the reported table, and beats STL on 5 of 11 tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ConsMTL instead of FairGrad | MTL on NYUv2 · NYUv2 | MR: 3.67 -> 2.78 | Helped |
| Use ConsMTL instead of FairGrad | MTL on NYUv2 · NYUv2 | ∆m%: -4.66 -> -6.72 | Helped |
| Use ConsMTL instead of Aligned-MTL | MTL on Cityscapes · Cityscapes | MR: 2.00 -> 4.00 | Hurt |
| Use ConsMTL instead of FairGrad | MTL on Cityscapes · Cityscapes | ∆m%: 5.18 -> -0.59 | Helped |
| Use ConsMTL instead of FairGrad | MTL on CelebA · CelebA | MR: 6.67 -> 3.40 | Helped |
| Use ConsMTL instead of FairGrad | MTL on CelebA · CelebA | ∆m%: 0.37 -> -1.42 | Helped |
| Use ConsMTL instead of Nash-MTL | MTL on QM9 · QM9 | MR: 4.55 -> 2.00 | Helped |
| Use ConsMTL instead of Nash-MTL | MTL on QM9 · QM9 | ∆m%: 4.55 -> 2.00 | Helped |
| Add lower-level optimization for task-specific parameters (ablation) | ConsMTL on QM9 · QM9 | gradient similarity | Helped |
| Increase λ from 0 toward a suitable intermediate value (ablation) | ConsMTL · NYUv2 and CelebA | ∆m% | Mixed |
| Set λ excessively large (ablation) | ConsMTL · NYUv2 and CelebA | ∆m% | Hurt |
Surprising findings
- Prior methods generally underperform STL on the surface-normal task on NYUv2, revealing task imbalance even among strong gradient-based methods.
Limitations
- Extra gradient computation adds memory and computational overhead.
- Hyperparameter λ requires task-specific tuning and its best value may change during training.
- Adaptive scheduling of λ is left for future work.
Experimental setup
Datasets: NYUv2 Cityscapes CelebA QM9
Benchmarks: NYUv2 Cityscapes CelebA QM9
Metrics: delta m% mean rank mIoU pixel accuracy absolute error relative error angle distance within t degrees MAE
Baselines: STL LS SI RLW DWA UW MGDA PCGrad GradDrop CAGrad IMTL-G MoCo
Builds on: MGDA Nash-MTL FairGrad FAMO Aligned-MTL
Closest papers by meaning
- Delve into the Applicability of Advanced Optimizers for Multi-Task Learning (NeurIPS 2026 · similarity 0.91)
- Selective Task Group Updates for Multi-Task Optimization (ICLR 2025 · similarity 0.90)
- Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learning (ICML 2024 · similarity 0.89)
- Fair Resource Allocation in Multi-Task Learning (ICML 2024 · similarity 0.88)
- MTLoRA: Low-Rank Adaptation Approach for Efficient Multi-Task Learning (CVPR 2024 · similarity 0.88)
- Quantifying Task Priority for Multi-Task Optimization (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.