Research map / Learning theory and algorithms / Prediction and decision losses
Robust Multi-Task Learning with Excess Risks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ExcessMTL, a multi-task weighting method that updates task weights by estimated excess risks (via a Taylor/empirical-Fisher approximation) instead of raw losses, to remain robust to label noise in some tasks.
Problem
Adaptive loss-based task weighting in multi-task learning over-weights tasks with label noise (high loss from noise rather than under-training), degrading performance on clean tasks.
Approach
Weight tasks by their estimated excess risk (distance to convergence) rather than loss, estimating excess risk locally with a second-order Taylor expansion and a diagonal empirical-Fisher Hessian approximation, updated by multiplicative weights (exponentiated gradient).
Key results
- On MultiMNIST with 80% noise on task 2, only ExcessMTL keeps clean-task performance close to the Bayes-optimal level, while loss-based methods degrade.
- On Office-Home with noise injected into the product task, ExcessMTL is least affected on clean tasks and is the only adaptive method assigning small weight to the noisy task.
- On NYUv2 with noise in semantic segmentation, ExcessMTL achieves the best performance on the clean tasks, while other methods fail under high label noise.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use excess-risk-based task weighting instead of loss-based adaptive weighting | ExcessMTL · MultiMNIST, clean task under increasing noise | clean-task accuracy | Helped |
| Use excess-risk-based task weighting instead of other adaptive weighting methods | ExcessMTL · Office-Home, clean tasks under increasing noise | clean-task performance | Helped |
| Use excess-risk-based task weighting instead of other adaptive weighting methods | ExcessMTL · NYUv2, depth estimation and surface normal estimation under semantic-segmentation noise | task performance | Helped |
| Increase label noise in one task while using adaptive weighting other than ExcessMTL | Adaptive weighting methods · MultiMNIST, clean task | clean-task performance | Hurt |
| Increase label noise in the product-classification task | Other adaptive methods · Office-Home | overall performance | Hurt |
| Increase noise in semantic segmentation | Other adaptive weighting algorithms · NYUv2, clean tasks | task performance | Hurt |
| Use uniform scalarization rather than adaptive weighting | Multi-task learning · MultiMNIST, noisy setting | clean-task performance | Mixed |
| Use uniform scalarization rather than adaptive weighting | Multi-task learning · Office-Home, noisy setting | overall performance | Hurt |
| Select task weights with MGDA on a clean validation set | MOML · Office-Home, low noise | performance | Hurt |
| Select task weights with MGDA on a clean validation set | MOML · Office-Home, high noise | overall performance | Hurt |
Surprising findings
- Uniform scalarization is resilient to label noise on MultiMNIST, though it underperforms adaptive methods on Office-Home.
- A clean validation set (MOML) does not fully alleviate the label noise issue because weight selection on clean validation still assigns considerable weight to noisy tasks.
Limitations
- Excess risk estimation relies on a local second-order Taylor approximation and diagonal empirical Fisher, which is approximate.
- Convergence guarantees rely on convexity assumptions; non-convex result is a stationarity bound with an unspecified approximation error sigma.
Experimental setup
Datasets: MultiMNIST Office-Home NYUv2
Benchmarks: MultiMNIST Office-Home NYUv2
Metrics: accuracy relative error angle accuracy pixel accuracy
Baselines: uniform scalarization GradNorm MGDA GroupDRO IMTL MOML
Builds on: GradNorm MGDA GroupDRO IMTL online mirror descent
Closest papers by meaning
- Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learning (ICML 2024 · similarity 0.86)
- Fair Resource Allocation in Multi-Task Learning (ICML 2024 · similarity 0.86)
- Theoretical Investigations and Practical Enhancements on Tail Task Risk Minimization in Meta Learning (NeurIPS 2024 · similarity 0.85)
- MTL-UE: Learning to Learn Nothing for Multi-Task Learning (ICML 2025 · similarity 0.85)
- NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspective (NeurIPS 2025 · similarity 0.85)
- Fast Rate Bounds for Multi-Task and Meta-Learning with Different Sample Sizes (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.