Research map / Continual learning / Continual fine-tuning of LLMs
Little By Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts
Haodong Lu, Chongyang Zhao, Jason Xue, Lina Yao, Kristen Moore, Dong Gong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace learned external router with self-activation over rank-1 atoms (ablation) | MoRAM · X-TAIL Last | accuracy (%): 69.76 -> 69.85 | No clear effect |
| use rank-1 atoms with external router instead of rank-r MoE-LoRA (ablation) | MoE-LoRA · X-TAIL Last | accuracy (%): 74.53 -> 69.76 | Hurt |
| add top-k sparsity to self-activation (ablation) | MoRAM · X-TAIL Last | accuracy (%): 69.85 -> 70.62 | Improved |
| add temperature scaling (tau_MoRAM) to self-activation (ablation) | MoRAM · X-TAIL Last | accuracy (%): 69.85 -> 79.62 | Improved |
| add threshold-based selection delta (ablation) | MoRAM · X-TAIL Last | accuracy (%): 69.85 -> 71.08 | Improved |
| combine top-k, temperature and threshold (ablation) | MoRAM · X-TAIL Last | accuracy (%): 69.85 -> 80.90 | Improved |
| add temperature scaling to coarse MoE-LoRA (ablation) | MoE-LoRA · X-TAIL Last | accuracy (%): 74.53 -> 74.57 | No clear effect |
| increase rank activation budget from 2 to 16 atoms (ablation) | MoRAM · 10-task checkpoint | accuracy (%) | Improved |
| use higher temperature tau_MoRAM (ablation) | MoRAM · X-TAIL Transfer | transfer accuracy | Improved |
| use MoRAM instead of LoRA r=32 | Llama-3.1-8B on CodeAlpaca · MMLU out-of-domain | accuracy (%): 63.59 -> 63.70 | Improved |
| use MoRAM instead of TreeLoRA | LLaMA-2-7B-Chat on TRACE · TRACE | BWT (%): 3.46 -> 1.37 | Improved |
| add continual learning with MoRAM over CLIP zero-shot (Transfer score) | CLIP on X-TAIL · X-TAIL Transfer | accuracy (%): 63.2 (CoDyRA) -> 63.3 | No clear effect |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- TreeLoRA — TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree (ICML 2025) · better on 3 of 3 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA (AAAI 2026 · similarity 0.90)
- PaLoRA: Paced Low-Rank Adaptation for Continual Learning (NeurIPS 2026 · similarity 0.89)
- Mixing Expertise with Confidence: A Mixture of Experts Framework for Robust Multi-Modal Continual Learning (ICML 2026 · similarity 0.89)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.89)
- Theory on Mixture-of-Experts in Continual Learning (ICLR 2025 · similarity 0.88)
- Divide and not forget: Ensemble of selectively trained experts in Continual Learning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
