Research map / Continual learning / Continual fine-tuning of LLMs
Continual Optimization with Symmetry Teleportation for Multi-Task Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COST, a plug-and-play multi-task learning method that uses LoRA-based symmetry teleportation to move the shared backbone to a loss-equivalent, less-conflicting point, combined with a historical trajectory reuse strategy for Adam-style optimizers.
Problem
Optimization conflict and task imbalance in multi-task learning with a shared backbone limit performance of optimization-based MTL methods.
Approach
Instead of reweighting losses or gradients, when dominated gradient conflict occurs the method teleports the shared backbone to another point on the same loss level set, found by training a LoRA adapter with a loss-invariance objective and a sharpness-based gradient-maximization objective, then merges it back; a historical trajectory reuse rule rescales optimizer moment estimates by the cosine bet
Key results
- On CityScapes, COST reduces Delta m% from 5.18 (FairGrad) to 4.30 and achieves the best MR among compared methods on NYUv2 (MR 3.22 vs 3.00 for FairGrad is not best; COST reports Delta m% -5.39 vs -4.66 for FairGrad).
- On CelebA (40-task), COST achieves MR 4.80, the best reported, and Delta m% 0.93 vs 1.15 for FairGrad-R.
- Plug-and-play: adding COST improves CAGrad's CityScapes Delta m% from 11.58 to 8.37 and FairGrad's from 5.18 to 4.30.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add COST to FairGrad | MTAN · CityScapes | ∆m%: 5.18 -> 4.30 | Helped |
| Add COST to FairGrad | MTAN · NYUv2 | ∆m%: -4.66 -> -5.39 | Helped |
| Add COST to the model | 9-layer CNN · CelebA | MR: 5.53 -> 4.80 | Helped |
| Add COST to the model | QM9 model · QM9 | MR: 4.27 -> 4.18 | Helped |
| Add COST to the model | QM9 model · QM9 | ∆m%: 58.5 -> 58.3 | Helped |
| Add loss invariance objective (Lt) (ablation) | COST on MTAN · CityScapes | ∆m%: 7.90 -> 4.30 | Helped |
| Add gradient maximization objective (Lg) (ablation) | COST on MTAN · CityScapes | ∆m%: 381.86 -> 4.30 | Helped |
| Add historical trajectory reuse (HTR) (ablation) | COST on MTAN · CityScapes | ∆m%: 4.65 -> 4.30 | Helped |
Surprising findings
- Ablation shows that using LoRA-based teleportation without the loss-invariance objective severely degrades performance (Delta m% 381.86), so loss invariance is critical.
Experimental setup
Datasets: CityScapes NYUv2 CelebA QM9
Benchmarks: CityScapes NYUv2 CelebA QM9
Metrics: mIoU pixel accuracy absolute error relative error Delta m% mean rank
Baselines: LS RLW DWA Uncertainty weighting MGDA GradDrop PCGrad CAGrad IMTL Nash-MTL FAMO FairGrad
Builds on: FairGrad LoRA Adam Sharpness-aware minimization
Closest papers by meaning
- Delve into the Applicability of Advanced Optimizers for Multi-Task Learning (NeurIPS 2026 · similarity 0.89)
- Improving Convergence and Generalization Using Parameter Symmetries (ICLR 2024 · similarity 0.88)
- Towards Consistent Multi-Task Learning: Unlocking the Potential of Task-Specific Parameters (CVPR 2025 · similarity 0.88)
- TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment (NeurIPS 2026 · similarity 0.86)
- Selective Task Group Updates for Multi-Task Optimization (ICLR 2025 · similarity 0.85)
- DaCe-DT: Data-Centric Offline Multi-Task Reinforcement Learning via Adaptive Prompts and Trajectory Correction for Heterogeneous Tasks (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.