Research map / Continual learning / Continual fine-tuning of LLMs
Continual Forgetting for Pre-trained Vision Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines continual forgetting for pre-trained vision models and proposes GS-LoRA, which uses per-task LoRA on FFN layers with group-sparse regularization to erase sequential forgetting requests while preserving remaining classes.
Problem
Removing specific learned knowledge (privacy-sensitive or wrong classes) from pre-trained vision models sequentially as erasure requests arrive over time, without harming retained classes or requiring costly retraining.
Approach
Fine-tune LoRA modules inserted into FFN layers of Transformer blocks for each forgetting task, with a group Lasso penalty over per-block LoRA groups to automatically select few blocks to modify, using a ReLU-bounded negative loss on forget data and a rehearsal loss on retained data.
Key results
- On face recognition (100-90 single-step), GS-LoRA reaches 72.2 H-Mean with 70.5 retained accuracy and 0.5 forgotten accuracy while tuning 1.28% of parameters.
- On COCO object detection (80-70 forgetting), GS-LoRA achieves 43.6 retained AP and 0.9 forgotten AP using 0.62% tunable parameters.
- In continual forgetting on face recognition (4 tasks, 20 classes each), GS-LoRA attains the best H-Mean among compared methods (71.6 at 100-20 setting) with forgotten accuracy of 3.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GS-LoRA instead of SCRUB-S | Face Transformer · CASIA-Face100, 100-50 | H: 54.8 -> 71.9 | Helped |
| use GS-LoRA instead of retraining | Deformable DETR · COCO, 80-1 | H: 46.6 -> 49.9 | Helped |
| use GS-LoRA instead of SCRUB-S | Face Transformer · CASIA-Face100, 100-20 continual | H: 71.2 -> 71.6 | Helped |
| add group sparse loss (ablation) | GS-LoRA · COCO, forget one class | APf: 0.40 vs. 0 | No clear effect |
| add group sparse loss (ablation) | GS-LoRA · COCO, forget one class | APr: 44.49 vs. 44.51 | No clear effect |
| add sparsity warm-up (ablation) | GS-LoRA, Face Transformer · Face Transformer, forget 10 classes | Accf: 73.78 -> 1.97 (α=0.01); 73.78 -> 0.70 (α=0.02) | Helped |
| vary LoRA rank from 2 to 8 (ablation) | GS-LoRA, Deformable DETR · COCO, forget 10 classes | APf: 7.60 -> 2.89 | Helped |
| vary LoRA rank from 8 to 32 (ablation) | GS-LoRA, Deformable DETR · COCO, forget 10 classes | APf: 2.89 -> 3.26 | Hurt |
| reduce training data ratio from 0.5 to 0.1 (ablation) | GS-LoRA, Face Transformer · Face Transformer, forget 10 classes | Accf: 0.93 -> 1.97 | Hurt |
| use a 12-block rather than 6-block model (ablation) | GS-LoRA, Face Transformer · Face Transformer, forget 10 classes | Accf: 1.97 -> 2.90 | Hurt |
| recover the model by fine-tuning the output FFN layer rather than relying on head forgetting (ablation) | GS-LoRA · Face Transformer recovery experiment | accuracy on forgotten classes: approximately 17% vs. 70% | Mixed |
Surprising findings
- Head-only masking of output FFN weights is easily recovered by fine-tuning, so backbone forgetting via GS-LoRA is argued to be more genuine.
- Without sparsity warm-up, forgetting fails entirely for larger group-sparsity coefficients because the model stays trapped in the original local minima.
Limitations
- Evaluated only on face recognition, COCO object detection and image classification; not yet on large language models.
- Forgetting is shown on relatively small-scale face and classification settings, and slight overfitting occurs for the 18-block model due to limited data.
- Recovery after backbone forgetting remains possible at a low accuracy level and harms retained classes.
Experimental setup
Datasets: CASIA-WebFace CASIA-Face100 COCO 2017
Benchmarks: CASIA-Face100 COCO 2017
Metrics: accuracy on retained classes accuracy on forgotten classes average precision H-Mean
Baselines: L2 regularization EWC MAS LwF DER DER++ FDR SCRUB SCRUB-S LIRF Retrain
Builds on: LoRA Elastic Weight Consolidation Memory Aware Synapses group Lasso Deformable DETR
Closest papers by meaning
- C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models (NeurIPS 2026 · similarity 0.86)
- Targeted Forgetting of Image Subgroups in CLIP Models (CVPR 2025 · similarity 0.85)
- Decoupled Distillation to Erase: A General Unlearning Method for Any Class-centric Tasks (CVPR 2025 · similarity 0.85)
- Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learning (NeurIPS 2024 · similarity 0.84)
- Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector (ICML 2025 · similarity 0.84)
- Black-Box Forgetting (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.