Research map / Label-efficient and robust learning / Recognition and long-tail

NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspective

NeurIPS 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Dense features and foundation priors · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes NTKMTL, which uses an extended neural tangent kernel analysis of multi-task training dynamics to set task weights that balance per-task NTK maximum eigenvalues, plus an efficient shared-representation variant NTKMTL-SR.

Problem

Task imbalance in multi-task learning, where some tasks dominate training and others are under-optimized; existing convergence-speed proxies based on loss differences or ratios fail to characterize task convergence accurately.

Approach

Extends NTK theory to MTL via an extended NTK matrix over all tasks, shows task weights scale the NTK, and sets weights omega_i = sqrt(mean eigenvalue / lambda_i) so each task's maximum NTK eigenvalue is balanced; NTKMTL-SR computes the NTK on the shared representation z, needing only one shared-parameter backward pass per iteration.

Key results

• On NYUv2, NTKMTL achieves mean rank 4.33 and Delta m% of -6.99, the best among compared methods.
• On CelebA (40 tasks), NTKMTL reaches Delta m% of -0.77, the only method to beat STL in that setting per the paper.
• On MT10, NTKMTL reaches 0.96 success rate, close to Aligned-MTL's 0.97 and STL's 0.90.

Measured effects reported in the paper

Surprising findings

• Reproducing baselines on QM9 with adjusted learning-rate scheduler and batch size substantially changed rankings and improved most baselines, suggesting prior QM9 comparisons were under-converged.

Limitations

• Weight design focuses only on diagonal task NTK blocks; off-diagonal extended NTK structure not exploited.
• NTKMTL-SR is hard to apply to multi-task RL because a single policy does not separate shared and task-specific parameters.
• Full NTK computation with n mini-batches is costly, so NTKMTL uses n=1 in experiments.

Experimental setup

Datasets: NYUv2 CityScapes QM9 CelebA MetaWorld MT10

Benchmarks: NYUv2 CityScapes QM9 CelebA MT10

Metrics: Delta m% mean rank mIoU pixel accuracy MAE success rate

Baselines: LS SI RLW DWA UW MGDA PCGrad GradDrop CAGrad IMTL-G Nash-MTL MoCo

Builds on: neural tangent kernel gradient flow Linear Scalarization Scale-Invariant loss Aligned-MTL

Closest papers by meaning

• Revisiting Fairness in Multitask Learning: A Performance-Driven Approach for Variance Reduction (CVPR 2025 · similarity 0.88)
• Towards Consistent Multi-Task Learning: Unlocking the Potential of Task-Specific Parameters (CVPR 2025 · similarity 0.88)
• MTLoRA: Low-Rank Adaptation Approach for Efficient Multi-Task Learning (CVPR 2024 · similarity 0.87)
• MTL-UE: Learning to Learn Nothing for Multi-Task Learning (ICML 2025 · similarity 0.87)
• Delve into the Applicability of Advanced Optimizers for Multi-Task Learning (NeurIPS 2026 · similarity 0.85)
• Identifying and Mitigating Spurious Correlation in Multi-Task Learning (CVPR 2025 · similarity 0.85)