Research map / Learning theory and algorithms / Matrix and tensor methods
Efficient Pareto Manifold Learning with Low-Rank Structure
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LORPMAN, which learns a Pareto manifold of networks as a shared main network plus per-task low-rank matrices with orthogonal regularization, cutting parameters versus PaMaL while improving hypervolume especially with many tasks.
Problem
Continuous Pareto front approximation for multi-task learning via linear combinations of base networks (PaMaL) scales poorly in parameters and convergence when the number of tasks is large.
Approach
Replace the m independent base networks of PaMaL with one main network plus m low-rank matrices per layer, combined by the preference vector, and add an orthogonality regularizer among the low-rank matrices so the main network captures shared features while the low-rank terms capture task-specific differences.
Key results
- On CIFAR-100 with 20 tasks, LORPMAN reaches HV 0.887 (x10^-2) versus 0.0583 for PaMaL, using 8.9% of PaMaL's parameters.
- On MultiMNIST, LORPMAN achieves HV 0.918 versus 0.905 for PaMaL and 0.906 for PHN-HVI.
- On CelebA with 40 tasks, LORPMAN achieves HV 1.167 (x10^-2) versus 0.472 for PaMaL using about 21% of its parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace PaMaL's task-specific base networks with a shared main network and low-rank matrices | LORPMAN · MultiMNIST | HV: 0.905 (0.0010) -> 0.918 (0.0018) | Helped |
| replace PaMaL's task-specific base networks with a shared main network and low-rank matrices | LORPMAN · CIFAR-100 | HV (× 10^-2): 0.0583 (0.010) -> 0.887 (0.047) | Helped |
| replace PaMaL's task-specific base networks with a shared main network and low-rank matrices | LORPMAN · CelebA | HV (× 10^-2): 0.472 (0.018) -> 1.167 (0.008) | Helped |
| add orthogonal regularization (ablation) | LORPMAN · UTKFace | HV: 0.309 (0.002) -> 0.314 (0.001) | Helped |
| increase rank from 4 to 64 (ablation) | LORPMAN · UTKFace | HV: 0.306 (0.002) -> 0.314 (0.001) | Helped |
| increase rank from 64 to 128 (ablation) | LORPMAN · UTKFace | HV: 0.314 (0.001) -> 0.313 (0.002) | Hurt |
| freeze the main module at epoch 80 instead of leaving it unfrozen (epoch 100) (ablation) | LORPMAN · UTKFace | HV: 0.307 (0.003) -> 0.314 (0.001) | Helped |
| set scaling factor s within 0.1–1 rather than s=2 (ablation) | LORPMAN · UTKFace | HV: 0.310 (0.002) to 0.314 (0.001) | No clear effect |
Surprising findings
- PaMaL performs very poorly on CIFAR-100 (HV 0.0583 x10^-2) and simply increasing its learning rate does not fix convergence.
- Orthogonal regularization reduces average correlation among low-rank matrices from 0.466 to 0.067 on UTKFace.
Limitations
- Uses the same rank for all layers, which the authors note as a limitation.
- Orthogonal regularization uses stochastic approximation with subsets of 3 tasks when m exceeds 3, which is an approximation.
- Approximation theorem is a universal-approximation style existence result and does not guarantee training finds it.
Experimental setup
Datasets: MultiMNIST Census UTKFace CIFAR-100 CelebA CityScapes
Benchmarks: MultiMNIST Census UTKFace CIFAR-100 CelebA CityScapes
Metrics: hypervolume number of parameters mIoU pixel accuracy absolute error relative error
Baselines: PHN PHN-HVI COSMOS PaMaL EPO PMTL MOO-SVGD PNG GMOOAR LS UW MGDA
Builds on: PaMaL COSMOS Pareto hypernetworks LoRA multi-forward regularization
Closest papers by meaning
- Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferences (ICLR 2025 · similarity 0.86)
- IOP: An Idempotent-Like Optimization Method on the Pareto Front of Hypernetwork (AAAI 2025 · similarity 0.86)
- StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold (NeurIPS 2025 · similarity 0.84)
- Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models (ICML 2024 · similarity 0.84)
- Towards Principled Task Grouping for Multi-Task Learning (NeurIPS 2026 · similarity 0.83)
- DMTG: One-Shot Differentiable Multi-Task Grouping (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.