Research map / Continual learning / Continual fine-tuning of LLMs
Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA
Zhan Fa, Yue Duan, Jian Zhang, Lei Qi, Wanqi Yang, Yinghuan Shi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add rank-1 expert pool (dynamic composition) to vanilla LoRA (ablation) | Vanilla LoRA · MTIL Order 1 | Transfer: 65.0 -> 69.7 | Improved |
| Add rank-1 expert pool to vanilla LoRA (ablation) | Vanilla LoRA · MTIL Order 1 | Last: 73.8 -> 77.1 | Improved |
| Add AGO loss to vanilla LoRA (ablation) | Vanilla LoRA · MTIL Order 1 | Transfer: 65.0 -> 65.3 | No clear effect |
| Add AGO loss to vanilla LoRA (ablation) | Vanilla LoRA · MTIL Order 1 | Average: 68.5 -> 69.2 | Improved |
| Add AGO loss on top of expert pool (ablation) | Expert pool LoRA · MTIL Order 1 | Last: 77.1 -> 77.9 | Improved |
| Merge Top-8 experts instead of Top-4 (ablation) | Ours · MTIL Order 1 | Last: 77.9 -> 73.7 | Hurt |
| Merge Top-2 experts instead of Top-4 (ablation) | Ours · MTIL Order 1 | Transfer: 70.3 -> 70.5 | No clear effect |
| Merge low-frequency experts (Down-4) instead of high-frequency (Top-4) (ablation) | Ours · MTIL Order 1 | Last: 77.9 -> 62.9 | Hurt |
| Merge Mid-4 instead of Top-4 experts (ablation) | Ours · MTIL Order 1 | Last: 77.9 -> 77.4 | Hurt |
| Increase AGO loss weight lambda (ablation) | Ours · MTIL | Transfer vs downstream accuracy | Mixed |
| Rank-1 expert pool single LoRA vs ZSCL | ZSCL · MTIL Order 1 task-ID free | Average: 64.4 -> 73.0 | Improved |
| Rank-1 expert pool vs ZSCL training cost | ZSCL · Training params | MB: 570.76 -> 18.99 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- SD-LoRA — SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning (ICLR 2025) · better on 3 of 3 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Shared LoRA Subspaces for almost Strict Continual Learning (ECCV 2026 · similarity 0.92)
- C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models (NeurIPS 2026 · similarity 0.92)
- PaLoRA: Paced Low-Rank Adaptation for Continual Learning (NeurIPS 2026 · similarity 0.92)
- TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree (ICML 2025 · similarity 0.91)
- SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning (ICLR 2025 · similarity 0.91)
- Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
