Research map / Optimization / Optimizers
Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Langevin unlearning: projected noisy gradient descent for learning and fine-tuning on the updated dataset, giving approximate Rényi unlearning guarantees that also cover non-convex ERM, sequential, and batch requests.
Problem
Approximate machine unlearning with privacy guarantees (certified indistinguishability from retraining) has only been established for (strongly) convex objectives, and existing methods need second-order updates or lack non-convex support and multi-request handling.
Approach
Treat learning as projected noisy gradient descent (Langevin dynamics) that converges to a unique stationary distribution; the learning RDP guarantee sets the initial divergence between learned and retrained distributions, and K PNGD unlearning iterations on the new dataset exponentially decay the Rényi difference ("privacy recuperation"). LSI constants are tracked along the process, with a k-inde
Key results
- Langevin unlearning with K=1 iteration significantly outperforms D2D in test accuracy for epsilon from 0.1 to 5 on MNIST and CIFAR10 and matches retraining accuracy.
- For unlearning 100 points with S=20 per request, Langevin unlearning saves about 40% unlearning iterations compared to D2D at the same (1,1/n)-unlearning guarantee.
- Retraining needs T=10,000 PNGD iterations while Langevin unlearning needs roughly up to K=2500 iterations for comparable utility.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Langevin unlearning instead of D2D with K=1 | Unlearning one data point on MNIST and CIFAR10 · MNIST and CIFAR10, one-point unlearning | test accuracy | Helped |
| Use Langevin unlearning instead of retraining from scratch | Unlearning one data point · MNIST and CIFAR10, one-point unlearning | test accuracy | No clear effect |
| Increase the unlearning batch size to S=20 | Langevin unlearning vs D2D · Unlearning 100 points | unlearning iterations: saves 40% unlearning iteration | Helped |
| Use a small batch size of S=5 | Langevin unlearning vs D2D · Unlearning 100 points | unlearning iterations | Hurt |
| Increase the noise level σ | Langevin unlearning · Batch unlearning of 100 points | unlearning efficiency | Helped |
| Increase the noise level σ | Langevin unlearning · Batch unlearning of 100 points | test accuracy | Hurt |
| Use a smaller noise level σ | Langevin unlearning · Batch unlearning of 100 points | unlearning iterations | Hurt |
| Increase the number of points unlearned per batch | Langevin unlearning · Batch unlearning | unlearning iterations | Hurt |
| Require a smaller unlearning privacy parameter ε | Langevin unlearning · Batch unlearning | unlearning iterations | Hurt |
| Vary batch size and privacy constraint | Langevin unlearning · Batch unlearning | test accuracy | No clear effect |
Surprising findings
- Langevin unlearning can be more expensive than D2D when unlearning only a small batch (S=5) due to the weak triangle inequality of Rényi divergence.
Limitations
- Non-convex privacy bound is loose due to undetermined constants and exponential dependence on hyperparameters, so experiments only cover strongly convex logistic regression.
- Analysis covers full-batch PNGD; the mini-batch SGD extension is left open.
- Small batch unlearning can cost more than D2D due to the weak triangle inequality for sequential requests.
Experimental setup
Datasets: MNIST CIFAR10
Benchmarks: MNIST CIFAR10
Metrics: test accuracy epsilon-delta unlearning privacy unlearning iterations
Baselines: Delete-to-Descent (D2D) retraining from scratch with PNGD Neel et al. PGD-based unlearning
Builds on: DP-SGD Delete-to-Descent Rényi differential privacy Langevin Monte Carlo analysis
Closest papers by meaning
- Leveraging Per-Instance Privacy for Machine Unlearning (ICML 2025 · similarity 0.94)
- Certified Machine Unlearning via Noisy Stochastic Gradient Descent (NeurIPS 2024 · similarity 0.94)
- The Utility and Complexity of In- and Out-of-Distribution Machine Unlearning (ICLR 2025 · similarity 0.94)
- Rewind-to-Delete: Certified Machine Unlearning for Nonconvex Functions (NeurIPS 2025 · similarity 0.92)
- Certified Unlearning for Neural Networks (ICML 2025 · similarity 0.92)
- Efficient Source-free Unlearning via Energy-Guided Data Synthesis and Discrimination-Aware Multitask Optimization (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.