Research map / Continual learning / Continual fine-tuning of LLMs
Continual Learning with Global Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes global alignment for continual learning by composing task-specific data representations as interpolations of shared pre-trained token representations (wiring models and controlled LoRA), combined with probing-then-fine-tuning, achieving strong no-replay CL results on BERT-base.
Problem
Catastrophic forgetting in continual learning caused by gradient interference between tasks, which depends on correlations between data representations and class vectors.
Approach
Learn each task's data representation as a task-specific interpolation (attention-based wiring) of pre-trained token representations so that cross-task representation correlations are grounded in pre-trained token correlations; add neighbor attention and a controlled-scale LoRA variant, plus a probing-then-fine-tuning initialization of class vectors.
Key results
- Wire-Fixed+PF reaches 90.18% average task-IL accuracy on the All sequence with 1.79 forgetting, versus 84.09 for IDBR(-R) and 88.89 for IDBR (with replay).
- On DB class-IL, alignment models approach ERACE, a replay-based class-incremental model, despite training only with task-incremental loss.
- C-LoRA+PF improves News Series accuracy from 74.83 to 78.59 over C-LoRA without PF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply probing then fine-tuning to Wire-Fixed (ablation) | Wire-Fixed · All Task-IL | ACC: 88.63\std{1.05} -> 90.18\std{0.31} | Helped |
| Apply probing then fine-tuning to Wire-Neigh (ablation) | Wire-Neigh · News Series Task-IL | ACC: 77.10\std{0.99} -> 77.90\std{0.45} | Helped |
| Apply probing then fine-tuning to C-LoRA (ablation) | C-LoRA · News Series Task-IL | ACC: 74.83\std{2.58} -> 78.59\std{0.85} | Helped |
| Increase Wire-Neigh scaling factor from 0.1 to 1.0 (ablation) | Wire-Neigh · News Series Task-IL | ACC: 77.10 -> 66.59 | Hurt |
| Increase C-LoRA scaling factor from 0.1 to 1.0 (ablation) | C-LoRA · News Series Task-IL | ACC: 74.83 -> 69.59 | Hurt |
| Increase C-LoRA+PF scaling factor from 0.1 to 1.0 (ablation) | C-LoRA+PF · News Series Task-IL | ACC: 78.59, 77.41, 76.83, 76.81 | Mixed |
| Increase Wire-Neigh neighborhood range K from 5 to 50 (ablation) | Wire-Neigh · News Series Task-IL | ACC: 76.90 -> 77.20 | Helped |
| Increase Wire-Neigh neighborhood range K from 50 to 100 (ablation) | Wire-Neigh · News Series Task-IL | ACC: 77.20 -> 76.58 | Hurt |
| Add neighbor attention to Wire-Fixed (ablation) | Wiring models · News Series Task-IL | ACC: 76.28 -> 77.10 | Helped |
| Use controlled scaling in LoRA rather than the original LoRA scaling | LoRA · Task-IL | ACC | Helped |
| Use global-alignment models instead of fine-tuning with replay | Continual learning models · Task-IL | forgetting | Helped |
| Use wiring models rather than adaptation baselines | Continual learning models · Task-IL | forgetting | Helped |
Surprising findings
- Probing-then-fine-tuning helps most when the alignment effect is weak (e.g., C-LoRA, FT) and gives little gain for strong wiring models.
- Wiring models without replay can outperform experience-replay CL models such as ER and A-GEM on task-incremental accuracy.
Limitations
- Without replay, models may suffer shifting attention where previously task-related tokens lose relevance to earlier tasks.
- Requires a well-established pre-trained model with semantic token representations, limiting applicability to domains without such models.
- Wire-Neigh adds memory and neighbor-search cost; efficiency improvements are left for future work.
Experimental setup
Datasets: Yahoo DBPedia AG_news MRPC RTE SST SNLI E-SNLI
Benchmarks: Yahoo DB News Series All
Metrics: average accuracy forgetting Recall@20
Baselines: Fine-tuning Adapter LoRA Prefix Tuning ER A-GEM MBPA++ IDBR CTR L2P CODA ERACE
Builds on: LoRA probing then fine-tuning IDBR MBPA++ A-GEM
Closest papers by meaning
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.89)
- Mitigating Catastrophic Forgetting in Online Continual Learning by Modeling Previous Task Interrelations via Pareto Optimization (ICML 2024 · similarity 0.89)
- Disentangling and mitigating the impact of task similarity for continual learning (NeurIPS 2024 · similarity 0.89)
- STAR: Stability-Inducing Weight Perturbation for Continual Learning (ICLR 2025 · similarity 0.89)
- A Unified and General Framework for Continual Learning (ICLR 2024 · similarity 0.88)
- The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting — An Analytical Model (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.