Research map / Continual learning / Online and replay-based
BrainWash: A Poisoning Attack to Forget in Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BrainWash, a norm-bounded data poisoning attack that uses model inversion and a first-order bilevel optimization to make a continual learner forget earlier tasks when it learns a poisoned task.
Problem
Continual learners are vulnerable to adversarial data poisoning that induces catastrophic forgetting of previously learned tasks, and the attacker typically lacks access to earlier tasks' data and the victim's CL algorithm.
Approach
Approximate previous-task data via model inversion on the victim's model, then craft l-infinity-bounded additive noise on the current task via a first-order bilevel optimization that maximizes forgetting on the inverted data, with an optional clean-accuracy term (cautious attacker) for the current task.
Key results
- On 10-split CIFAR-100 with EWC and epsilon=0.3, the reckless BrainWash attack lowers BWT from -5.2 (clean) to -29.1 while last-task accuracy falls to 25.5%.
- Against replay-based ER on 10-split CIFAR-100, BrainWash with model inversion reduces BWT from -18.9 to -23.1 (accuracy 71.2 to 53.8).
- Regularized model inversion achieves forgetting of 28.16% vs 28.8% with real previous-task data, versus 23.6% without regularization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Poison the final task with BrainWash instead of training it cleanly | EWC continual learner · 10-Split CIFAR-100 | BWT: -5.2 -> -29.1 | Helped |
| Poison the final task with BrainWash instead of training it cleanly | EWC continual learner · 10-Split miniImageNet | BWT: -3.9 -> -34.4 | Helped |
| Poison the final task with BrainWash instead of training it cleanly | EWC continual learner · 20-Split tinyImageNet | BWT: -0.4 -> -25.9 | Helped |
| Use cautious rather than reckless poisoning | EWC on CIFAR-100 · 10-Split CIFAR-100 | BWT: -24.7 vs -29.1 | Mixed |
| Increase the poisoning noise bound | BrainWash on CIFAR-100, miniImageNet, and tinyImageNet · Continual learning benchmarks | BWT: See Table 1 (ϵ = 0.1 and ϵ = 0.3) | Helped |
| Read the victim's replay memory while applying BrainWash | ER replay-based CL · 10-Split CIFAR-100 | BWT: -23.1 -> -27.7 | Helped |
| Use BrainWash rather than Unlearnable Examples, DeepConfuse, or Meta-Poison | EWC · 10-Split CIFAR-100 | BWT: -21.4, -19.42, -22.94 -> -29.1 | Helped |
| Increase the regularization coefficient λ (ablation) | BrainWash with EWC · 10-Split CIFAR-100 | forgetting | Hurt |
| Increase the noise injection rate or noise magnitude (ablation) | BrainWash with EWC · CIFAR-100, five tasks | forgetting | Helped |
| Remove regularization from model inversion (ablation) | BrainWash with EWC · 10-Split CIFAR-100 | forgetting: 28.8 -> 23.6 | Hurt |
| Replace inverted prior-task data with real prior-task data (ablation) | BrainWash with EWC · 10-Split CIFAR-100 | forgetting: 28.8 -> 28.16 | No clear effect |
| Apply BrainWash later in continual training, from 10 to 20 tasks (ablation) | BrainWash with EWC · CIFAR-100, five-way tasks | forgetting: 17.47 -> 14.04 | Hurt |
Surprising findings
- Higher EWC/regularization strength only marginally reduces attack effectiveness while severely hurting victim performance, so high stability is not an effective defense.
- ANCL and RWALK are notably more robust to the attack than EWC, AFEC, and MAS in the miniImageNet and tinyImageNet results.
Limitations
- Attack evaluated on ResNet-18 and RegNetX with image classification benchmarks only.
- Cautious attacker trades off forgetting strength against current-task accuracy.
- Attacker assumed to have access to the victim's current model and the clean data of the newest task.
Experimental setup
Datasets: CIFAR-100 miniImageNet tinyImageNet
Benchmarks: 10-Split CIFAR-100 10-Split miniImageNet 20-Split tinyImageNet
Metrics: backward transfer (BWT) accuracy on last task
Baselines: Uniform noise Unlearnable Examples Deep-Confuse Meta-Poison ER ER-ACE
Builds on: Meta-Poison Deep-Confuse Yin et al. DeepInversion Elastic Weight Consolidation Unlearnable Examples
Closest papers by meaning
- Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspective (ICML 2025 · similarity 0.87)
- STAR: Stability-Inducing Weight Perturbation for Continual Learning (ICLR 2025 · similarity 0.87)
- Predicting the Susceptibility of Examples to Catastrophic Forgetting (ICML 2025 · similarity 0.86)
- Towards Understanding Catastrophic Forgetting in Two-layer Convolutional Neural Networks (ICML 2025 · similarity 0.86)
- Attack on Prompt: Backdoor Attack in Prompt-Based Continual Learning (AAAI 2025 · similarity 0.86)
- Do Your Best and Get Enough Rest for Continual Learning (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.