Research map / Label-efficient and robust learning / Domain adaptation and generalization
Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoLA, a mixture of low-rank adapters jointly trained with a shared backbone to mitigate conflicts in heterogeneous multi-task/multi-domain training, with MoLA-Grad (task-ID routing) and MoLA-Router (router regularized by a Task-wise Decorrelation loss).
Problem
Training conflicts when a unified model is trained jointly on heterogeneous data from different domains or tasks, where hard/soft parameter sharing either lacks task-specific capacity or scales poorly in parameters.
Approach
Attach multiple low-rank adapters in parallel to a shared backbone, trained end-to-end from scratch; task identifiers (MoLA-Grad) or a router supervised by a Task-wise Decorrelation loss (MoLA-Router) assign task-specific adapter combinations so gradients are separated across tasks.
Key results
- On VLCS, MoLA-Grad reaches 86.41 average accuracy and MoLA-Router 86.40, versus 84.98 for HPS and 84.60 for single-task.
- On NYUv2, MoLA-Grad achieves a task-weighted Δm% of -9.17 versus -6.56 for the best prior MTL baseline (MTAN).
- On MedMNISTV2, MoLA-Router reaches 85.53 average accuracy, above the single-task model at 84.93.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MoLA-Grad (task-identifier-based low-rank adapters) vs HPS | ResNet multi-task backbone · VLCS | Avg accuracy: 86.41 vs 84.98 | Helped |
| Add MoLA-Grad (task-identifier-based low-rank adapters) | ResNet multi-task backbone · NYUv2 | Δm% (lower better): -9.17 vs 4.72 for HSP | Helped |
| Add MoLA-Router (TwD-regularized router) on ResNet | multi-task backbone · Officehome | Avg accuracy: 83.84 vs 78.18 for HPS | Helped |
| Add MoLA (both variants) on top of backbone for heterogeneous training | ResNet · MedMNISTV2 | Avg accuracy: 85.53 / 85.47 vs 84.19 for HSP | Helped |
| Use MoLA-Grad or MoLA-Router rather than uniform single shared network | ResNet · RadImageNet | Avg accuracy: 56.47 vs 38.49 for Uniform | Helped |
| Increase router rank from r=4 to r=8 or r=1 (ablation) | MoLA-Router · VLCS | Avg accuracy: 86.40 at r=4 vs 83.50 at r=8 and 83.80 at r=1 | Hurt |
| Increase adapter rank beyond r=4 (r=8,16,64) (ablation) | MoLA-Grad · VLCS | Avg accuracy: 86.41 at r=4 vs 84.35 at r=8, 84.53 at r=64 | Hurt |
| Apply MoLA to the transformer backbone instead of ResNet (ablation) | MoLA-Router and MoLA-Grad · Officehome and VLCS | Avg accuracy: Officehome Router 89.06 vs 83.84 (ResNet); VLCS Grad 89.40 vs 86.41 | Helped |
| Share a single router across blocks instead of non-shared routers (ablation) | MoLA-Router · Officehome (0+3 config) | Avg accuracy: 80.78 shared vs 75.41 non-shared | Helped |
| Apply MoLA to all four residual blocks instead of only Block4 (ablation) | MoLA-Grad · NYUv2 | Δm% (lower better): -1.30 for all blocks vs -9.17 for Block4 only | Hurt |
| Add MoLA at Block4 only (deep layers) (ablation) | MoLA-Grad · NYUv2 | Δm% (lower better): -9.17 vs 4.72 without MoLA | Helped |
| Increase number of adapters/experts in MoLA-Router from 2 to 4 (ablation) | MoLA-Router · VLCS and Officehome | Avg accuracy: VLCS n=3 86.00 best; Officehome n=4 81.29 best vs n=2 74.60 | Mixed |
Surprising findings
- A uniform single network can underperform single-task training on domain heterogeneous data (e.g., VLCS uniform 83.39 vs single-task 84.60).
- Increasing LoRA rank or number of adapters does not monotonically improve performance, which the authors attribute to over-fitting.
Limitations
- MoLA-Grad requires task identifiers at test time, limiting target-agnostic use.
- Performance is sensitive to rank, number of adapters, placement of MoLA, and router feature dimension, with no universal best setting.
Experimental setup
Datasets: VLCS Officehome RadImageNet MedMNISTV2 NYUv2
Benchmarks: VLCS Officehome RadImageNet MedMNISTV2 NYUv2
Metrics: classification accuracy average accuracy mIoU Pix Acc depth abs/rel error surface normal angle error task-weighted delta m%
Baselines: Single-Task Uniform HPS Cross-Stitch MMoE CGC LTB DSelect-K PCGrad CAGrad MGDA MMD-AAE
Builds on: LoRA Mixture of Experts MMoE CGC Aligned-MTL
Closest papers by meaning
- Sparse High Rank Adapters (NeurIPS 2024 · similarity 0.85)
- MTLoRA: Low-Rank Adaptation Approach for Efficient Multi-Task Learning (CVPR 2024 · similarity 0.85)
- Ensembles of Low-Rank Expert Adapters (ICLR 2025 · similarity 0.85)
- Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferences (ICLR 2025 · similarity 0.85)
- CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning (CVPR 2025 · similarity 0.85)
- MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.