Research map / Learning theory and algorithms / Combinatorial and search optimization
Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PaLoRA, a parameter-efficient Pareto Front Learning method that adds task-specific low-rank adapters to any network and uses an annealed deterministic preference sampling schedule to trace the Pareto front.
Problem
Pareto Front Learning (PFL) methods are memory-hungry, converge slowly, and produce inconsistent mappings from preference vectors to objective space due to random preference sampling.
Approach
Augment each layer with task-specific low-rank adapters weighted by the preference vector (so the base network learns shared features and adapters learn task-specific ones), and replace random Dirichlet preference sampling with a deterministic, evenly spaced, temperature-annealed schedule that moves from simplex center to edges over training.
Key results
- On Cityscapes, PaLoRA reaches 71.11 mIoU and 92.21 pixel accuracy with 4.2% parameter increase, versus PaMaL at 70.35 mIoU and 100% increase.
- On NYU-v2, PaLoRA achieves 38.27 segmentation mIoU (best among methods) with 6.3% parameter increase, while PaMaL requires 200%.
- On MultiMNIST, PaLoRA (rank 1) reaches 96.55 top-left and 95.39 bottom-right accuracy, above PaMaL's 96.17 and 95.32.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PaLoRA instead of PaMaL | SegNet multi-task PFL · Cityscapes test | Rel Err: 54.520 -> 51.2672 | Helped |
| Use PaLoRA instead of PaMaL | SegNet multi-task PFL · NYU test | mIoU: 33.94 -> 38.27 | Helped |
| Use PaLoRA instead of PaMaL | SegNet multi-task PFL · NYU test | Abs Err: 0.5592 -> 0.5370 | Hurt |
| Replace random preference sampling with deterministic annealed sampling | PaLoRA · MultiMNIST validation Pareto front | number of Pareto-front points | Helped |
| Use deterministic rather than random preference sampling (ablation) | PaLoRA · MultiMNIST | Spearman correlation | Helped |
| Vary LoRA rank and forward-pass count across the reported settings (ablation) | PaLoRA on SegNet · Cityscapes | mIoU: 71.44 to 71.64 | No clear effect |
| Increase LoRA rank from 1 to 8 at M=5 (ablation) | PaLoRA on SegNet · Cityscapes | Rel Err: 48.7060 -> 48.9209 | Mixed |
Surprising findings
- Pseudo-preferences outside the simplex (e.g., (0,0)) reveal that the base network holds general features while adapters hold task-specific ones, with negating one task's adapter causing forgetting of that task.
Limitations
- Evaluation of PFL methods requires a grid of preference points, making evaluation cost scale linearly with number of points.
- Gradient balancing used for scene benchmarks makes runtime long (about 4.5 hours for NYU with M=1).
Experimental setup
Datasets: MultiMNIST MultiMNIST-3 UTKFace SARCOS Cityscapes NYU-v2
Benchmarks: MultiMNIST Cityscapes NYU-v2 UTKFace SARCOS MultiMNIST-3
Metrics: hypervolume mIoU pixel accuracy absolute error relative error surface normal angle distance parameter overhead
Baselines: LS UW MGDA DWA PCGrad IMTL CAGrad Nash-MTL RLW Graddrop RotoGrad Auto-lambda
Builds on: LoRA PaMaL Pareto Hypernetworks Linear Scalarization
Closest papers by meaning
- Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task models (ICML 2023 · similarity 0.88)
- Efficient Pareto Manifold Learning with Low-Rank Structure (ICML 2024 · similarity 0.86)
- MTLoRA: Low-Rank Adaptation Approach for Efficient Multi-Task Learning (CVPR 2024 · similarity 0.86)
- TADFormer: Task-Adaptive Dynamic TransFormer for Efficient Multi-Task Learning (CVPR 2025 · similarity 0.85)
- Sparse High Rank Adapters (NeurIPS 2024 · similarity 0.85)
- Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapters (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.