Research map / Continual learning / Online and replay-based
Privacy-Aware Lifelong Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PALL, a fixed-capacity lifelong learning method that trains task-specific sparse subnetworks with parameter sharing, enabling exact task unlearning via parameter reset and DER++-style episodic memory rehearsal.
Problem
Lifelong (task-incremental) learning needs to prevent catastrophic forgetting and allow forward transfer while also supporting exact unlearning of whole tasks under right-to-be-forgotten requests, with limited memory and no access to prior data.
Approach
Jointly optimize task-specific sparse subnetwork masks (via learned importance scores with straight-through estimator) within one fixed architecture, freeze used weights to stop forgetting, allow sharing of frozen weights for forward transfer, and exactly unlearn a task by resetting its trained parameters and repairing overlapping weights for later tasks with a rehearsal buffer.
Key results
- On S-CIFAR100, PALL (alpha=0.1) reaches 72.35 average accuracy on remaining tasks with zero learning forgetting, versus 73.22 for independent models with exact unlearning and 66.84 for DER++ (inexact).
- On S-TinyImageNet with 100 tasks and N_u=3, PALL (alpha=0.025) achieves 86.11 A_l with F_u 0.43, vs 86.80 for independent models and 83.75 for dynamic-sparse independent subnetworks.
- PALL uses roughly 4.3x to 12x less model size than training independent per-task models while providing exact unlearning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| retrain reset parameters using episodic memory rather than only resetting them (ablation) | PALL · S-CIFAR100 | average test accuracy of learned tasks (Al): 70.24 -> 72.35 | Helped |
| increase rehearsal retraining from 50 to 100 iterations (ablation) | PALL · S-CIFAR100 | average test accuracy of learned tasks (Al): 72.35 -> 72.46 | Helped |
| use DER++-style logit distillation (beta=0.5) instead of vanilla replay (beta=0.0) (ablation) | PALL · S-CIFAR100, buffer size 500 | average test accuracy of learned tasks (Al): 71.98 -> 72.35 | Helped |
| increase logit-distillation weight from beta=0.5 to beta=1.0 (ablation) | PALL · S-CIFAR10, buffer size 200 | average test accuracy of learned tasks (Al): 93.95 -> 93.36 | Hurt |
| increase episodic memory buffer size from 200 to 500 (ablation) | PALL · S-CIFAR100 | average test accuracy of learned tasks (Al): 71.90 -> 72.35 | Helped |
| increase episodic memory buffer size from 500 to 1000 (ablation) | PALL · S-CIFAR100 | average test accuracy of learned tasks (Al): 72.35 -> 73.58 | Helped |
| replace random buffer sampling with prioritized exemplar sampling (ablation) | PALL · S-CIFAR10 and S-CIFAR100 | average test accuracy of learned tasks (Al): 94.25 -> 94.15; 72.11 -> 72.10 | Mixed |
| remove knowledge transfer between task subnetworks (ablation) | PALL · S-TinyImageNet (T=100) | average test accuracy of learned tasks (Al): 83.75 -> 86.11 | Hurt |
| use dynamic rather than static sparse independent subnetworks (ablation) | independent subnetworks · S-TinyImageNet (T=100) | average test accuracy of learned tasks (Al): 80.70 -> 83.75 | Helped |
Surprising findings
- Simply resetting affected parameters without retraining (N_f=0) degrades performance noticeably, showing rehearsal retraining is necessary after unlearning.
- Prioritized exemplar selection (entropy, confidence, logit distance, herding) did not improve over random sampling for the rehearsal buffer and often hurt performance.
Limitations
- Only task-incremental setting with task IDs available; not directly applicable to class- or domain-incremental learning.
- Unlearning is complete per-task only; not per-sample deletion, and not combined with differential privacy.
- Evaluated only on image classification benchmarks (CIFAR and TinyImageNet), not language tasks.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 TinyImageNet
Benchmarks: Sequential CIFAR-10 Sequential CIFAR-100 Sequential TinyImageNet
Metrics: A_l accuracy on remaining tasks A_u accuracy on unlearned tasks F_l forgetting from learning F_u forgetting from unlearning
Baselines: Sequential EWC LwF LSF GEM ER DER++ PackNet WSN Independent
Builds on: PackNet winning subnetworks (WSN) DER++ SISA
Closest papers by meaning
- Certified Unlearning for Neural Networks (ICML 2025 · similarity 0.87)
- Leveraging Per-Instance Privacy for Machine Unlearning (ICML 2025 · similarity 0.86)
- Efficient Source-free Unlearning via Energy-Guided Data Synthesis and Discrimination-Aware Multitask Optimization (ICML 2025 · similarity 0.86)
- Fast Machine Unlearning without Retraining through Selective Synaptic Dampening (AAAI 2024 · similarity 0.86)
- Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearning (NeurIPS 2024 · similarity 0.86)
- The Unseen Threat: Residual Knowledge in Machine Unlearning under Perturbed Samples (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.