Research map / Optimization / Optimizers
Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Models machine unlearning as a constrained problem (maximize unlearning decrease under a bounded retain-loss increase), shows it equals unilateral gradient surgery, and approximates the multiplier with a finite difference so only one backprop per step is needed (EUPMU).
Problem
Machine unlearning in pre-trained generative and classification models must trade off forgetting targeted data against preserving utility; linear scalarization yields extreme solutions, and standard multi-objective methods under-optimize the unlearning objective while doubling compute.
Approach
Formulate utility preservation as a per-step constraint on retain-loss increase, derive a closed-form unilateral gradient surgery on the unlearning gradient, and approximate the Lagrange multiplier with the observed retain-loss change so the update needs only one backpropagation of the composite loss.
Key results
- On SD class-wise forgetting on Imagenette, EUPMU reaches 0.00 average unlearning accuracy versus 59.76 for FMN, with average FID 1.03 (best).
- On CIFAR-10 random-labeling ResNet-18 forgetting, EUPMU achieves average score 98.40 versus 97.48 for FAMO and 98.12 for UNGrad, at RTE 18.4 vs 27.7 for PCGrad/UNGrad.
- On DDPM class-wise forgetting on CIFAR-10, EUPMU attains FID 22.57 (best among ESD and SalUn) with RTE 1.38 vs 6.05 for SalUn.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace linearization with EUPMU | DDPM · CIFAR-10 Airplane forgetting | FID: 24.53 -> 22.57 | Helped |
| replace SalUn with EUPMU | Stable Diffusion · Imagenette class-wise forgetting average | UA: 0.18 -> 0.00 | Helped |
| replace SalUn with EUPMU | Stable Diffusion · Imagenette class-wise forgetting average | FID: 1.22 -> 1.03 | Helped |
| replace SalUn with EUPMU | Stable Diffusion · NSFW removal (i2p, total) | NSFW contents generated: 13 -> 6 | Helped |
| replace FAMO with EUPMU | ResNet 18 · CIFAR-10 class-wise forgetting (10%, random labeling) | Avg. score: 97.48 -> 98.40 | Helped |
| replace UNGrad with implicit gradient surgery in EUPMU | ResNet 18 · CIFAR-10 class-wise forgetting (10%, random labeling) | Avg. score: 98.12 -> 98.40 | Helped |
| replace UNGrad with implicit gradient surgery in EUPMU | ResNet 18 · CIFAR-10 class-wise forgetting (10%, random labeling) | RTE: 27.7 -> 18.4 | Helped |
| replace EUPMU with EUPMU-fast, using consecutive retain batches and removing the second retain-loss recomputation (ablation) | ResNet 18 · CIFAR-10 class-wise forgetting (10%, random labeling) | Avg. score: 98.40 -> 98.09 | Hurt |
| replace EUPMU with EUPMU-fast, using consecutive retain batches and removing the second retain-loss recomputation (ablation) | ResNet 18 · CIFAR-10 class-wise forgetting (10%, random labeling) | RTE: 18.4 -> 17.9 | Helped |
| replace ESD with EUPMU | Stable Diffusion v1.4 · Snoopy instance erasure | CS: 49.27 -> 44.9 | Helped |
| replace SPM with EUPMU | Stable Diffusion v1.4 · Van Gogh style erasure | FID: 23.64 -> 22.96 | Helped |
Surprising findings
- The approximate multiplier update (finite-difference retain-loss change) outperforms the exact explicit unilateral gradient surgery (UNGrad) on every reported metric, which the authors attribute to stochastic-training stability.
Limitations
- Theoretical guarantees rely on strong assumptions (convexity, boundedness, smoothness) that do not hold for deep networks; convergence analysis for the approximated multiplier is only sketched.
- EUPMU-fast's stochastic proxy sometimes underperforms EUPMU due to between-batch variation.
- Experiments are limited to small-scale image classification and diffusion concept erasure; no LLM experiments.
Experimental setup
Datasets: CIFAR-10 Imagenette i2p prompts
Benchmarks: CIFAR-10 Imagenette i2p
Metrics: unlearning accuracy FID CLIP score CLIP accuracy membership inference attack remaining accuracy testing accuracy runtime efficiency
Baselines: SalUn ESD FMN ConAbl SPM FAMO PCGrad Linearization UNGrad
Builds on: MGDA PCGrad FAMO SalUn
Closest papers by meaning
- When to Forget? Complexity Trade-offs in Machine Unlearning (ICML 2025 · similarity 0.90)
- The Utility and Complexity of In- and Out-of-Distribution Machine Unlearning (ICLR 2025 · similarity 0.89)
- Machine Unlearning under Overparameterization (NeurIPS 2025 · similarity 0.88)
- Efficient Source-free Unlearning via Energy-Guided Data Synthesis and Discrimination-Aware Multitask Optimization (ICML 2025 · similarity 0.88)
- Certified Machine Unlearning via Noisy Stochastic Gradient Descent (NeurIPS 2024 · similarity 0.88)
- DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.