Research map / Continual learning / Continual fine-tuning of LLMs
Exploiting Presentative Feature Distributions for Parameter-Efficient Continual Learning of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an online, information-leakage-free continual learning method for LLMs that routes instances to LoRA blocks via similarity to per-block presentative feature distributions computed from frozen pretrained features.
Problem
Continual learning of LLMs with PEFT typically relies on task IDs at test time or replay of past task data, causing information leakage (IL) that is incompatible with online, privacy-sensitive deployment.
Approach
Characterize each task's LoRA block by a presentative feature distribution (the mean of frozen-weight hidden features over the task data), then dynamically select and merge LoRA blocks per instance via softmax-normalized similarity (negative L2 or dot product), introducing no new trainable selection parameters and thus no new forgetting.
Key results
- With LLaMA-2-7B on SuperNI, Ours-L2 reaches 55.60 AP without information leakage, about 12.85 points above the best no-IL baseline TaSL (42.00), and only 0.63 below SAPT-LoRA which uses IL.
- Across both benchmarks with LLaMA-2-7B, Ours-L2 achieves 69.31 average performance, the best among methods without IL, surpassing SAPT-LoRA with IL (68.99).
- With LLaMA-2-13B, Ours-L2 achieves 70.61 average performance, higher than the 7B result, and the method adds only 0.03% parameters for LLaMA-2-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use L2 distance rather than dot-product similarity for selecting LoRA blocks (ablation) | Ours · SuperNI and Long Sequence Benchmarks | AP | Helped |
| Reduce Top-K in dynamic selection (ablation) | Ours-Dot · SuperNI and Long Sequence Benchmarks | AP | Helped |
| Use more model layers for feature distribution analysis (ablation) | Ours-L2 · SuperNI and Long Sequence Benchmarks | AP | Helped |
| Expand continual SuperNI with independently trained Long Sequence LoRAs and feature distributions (ablation) | Ours · SuperNI Benchmark | AP: 54.33 -> 53.70 (-0.63) | Hurt |
| Expand continual SuperNI with independently trained Long Sequence LoRAs and feature distributions (ablation) | Ours · Long Sequence Benchmark | AP: 29.41 -> 72.78 (+43.37) | Helped |
| Expand continual Long Sequence with independently trained SuperNI LoRAs and feature distributions (ablation) | Ours · SuperNI Benchmark | AP: 28.75 -> 52.12 (+23.37) | Helped |
| Expand continual Long Sequence with independently trained SuperNI LoRAs and feature distributions (ablation) | Ours · Long Sequence Benchmark | AP: 79.82 -> 78.49 (-1.33) | Hurt |
| Expand continual SuperNI with independently trained Long Sequence LoRAs and feature distributions (ablation) | Ours · SuperNI Benchmark | AP: 56.52 -> 55.78 (-0.74) | Hurt |
| Expand continual SuperNI with independently trained Long Sequence LoRAs and feature distributions (ablation) | Ours · Long Sequence Benchmark | AP: 31.93 -> 78.10 (+46.17) | Helped |
| Expand continual Long Sequence with independently trained SuperNI LoRAs and feature distributions (ablation) | Ours · SuperNI Benchmark | AP: 37.09 -> 54.88 (+17.79) | Helped |
| Expand continual Long Sequence with independently trained SuperNI LoRAs and feature distributions (ablation) | Ours · Long Sequence Benchmark | AP: 82.53 -> 81.73 (-0.80) | Hurt |
| Increase the pretrained model from LLaMA-2-7B to LLaMA-2-13B | Ours-L2 · SuperNI and Long Sequence Benchmarks | AP | Helped |
Surprising findings
- Methods without information leakage lose 12% to 72% performance relative to their IL counterparts (e.g., SAPT 82.02 vs 10.18 on Long Sequence in one setting), attributed to forgetting introduced by trainable query projections in the selection stage.
- L2 distance selection outperforms dot-product similarity, with dot-product similarity values being too close to each other to discriminate blocks.
Limitations
- Selection relies on statistics of frozen features, so performance depends on the pretrained model's representation quality; gains grow with model size.
- Dot-product similarity values are too close to distinguish blocks, making the method sensitive to the similarity choice and top-K setting.
Experimental setup
Datasets: SuperNI Benchmark Long Sequence Benchmark
Benchmarks: SuperNI Benchmark Long Sequence Benchmark
Metrics: Average Performance (AP) Forgetting Rate (F.Ra) Forward Transfer (FWT) Backward Transfer (BWT)
Baselines: Replay ProgPrompt SAPT-LoRA SeqLoRA LFPT5 EPI O-LoRA TaSL-LoRA
Builds on: LoRA SAPT O-LoRA TaSL ProgPrompt
Closest papers by meaning
- On Large Language Model Continual Unlearning (ICLR 2025 · similarity 0.87)
- Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.86)
- Quantifying and Analyzing Entity-Level Memorization in Large Language Models (AAAI 2024 · similarity 0.86)
- CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Models (AAAI 2025 · similarity 0.86)
- Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models (NeurIPS 2025 · similarity 0.85)
- Learning and Forgetting Unsafe Examples in Large Language Models (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.