Research map / Continual learning / Continual fine-tuning of LLMs

Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models

NeurIPS 2025 · Continual learning › Continual fine-tuning of LLMs · approach: Low-rank and model-merging adaptation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GainLoRA, a continual learning method for LLMs that expands a LoRA branch per task and adds per-task gating modules, with initialization and gradient-projection constraints that keep the new gate near zero on old-task inputs, reducing forgetting without task IDs or replay.

Problem

Continual learning of LLMs with LoRA: existing expansion-based LoRA methods add new branches with fixed integration coefficients of 1, so new branches equally affect old tasks and cause forgetting, in a setting without task identities at inference and without replay data.

Approach

Integrate LoRA branches via input-dependent gating coefficients produced by a per-task gating module; constrain the new gate's initialization and updates to be orthogonal to old-task input subspaces (via gradient projection memory) so its output on old-task inputs stays near zero, while leaving the new task's gate free to be near one.

Key results

• With T5-Large on Order 1, GainLoRA(O-LoRA) reaches AP 47.84 and FT 2.26 versus O-LoRA's 26.37 AP and 19.15 FT.
• On Llama-2-7B Order 1, GainLoRA(InfLoRA) achieves AP 51.27 and FT 2.84, compared with InfLoRA's 42.93 AP and 11.23 FT.
• GainLoRA(InfLoRA) with T5-XL on Order 3 reaches AP 81.22 and FT 0.58, improving on InfLoRA (80.22 AP, 2.09 FT).

Measured effects reported in the paper

Surprising findings

• Removing the initialization or updating constraints from GainLoRA degrades performance substantially, e.g. with T5-Large Order 1, removing both gives AP 26.32 versus 47.84 for the full method.

Limitations

• The imposed constraints may accumulate with increasing tasks and hinder learning of new tasks.
• Evaluated primarily on non-overlapping task sequences; overlapping-task scenarios are not studied.
• Gating prevents merging LoRA branches at inference, adding some FLOPs over methods that merge.

Experimental setup

Datasets: SuperNI Long Sequence benchmark (Razdaibiedina et al. CL benchmark, GLUE, SuperGLUE)

Benchmarks: SuperNI (Order 1, Order 2) Long Sequence benchmark (Order 3, Order 4)

Metrics: average performance (AP) forgetting (FT) Rouge-L accuracy

Baselines: LFPT5 EPI MIGU EWC TaSL KIFLoRA IncLoRA C-LoRA O-LoRA InfLoRA SeqLoRA

Builds on: LoRA O-LoRA InfLoRA gradient projection memory

Closest papers by meaning

• Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.92)
• CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning (CVPR 2025 · similarity 0.90)
• Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.90)
• Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.90)
• LoRA vs Full Fine-tuning: An Illusion of Equivalence (NeurIPS 2025 · similarity 0.89)
• SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning (ICLR 2025 · similarity 0.89)