Research map / Federated learning and privacy / Privacy attacks and secure learning
Certified Unlearning for Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes certified approximate unlearning for non-convex neural networks by running noisy, clipped fine-tuning on the retain set, analyzed via privacy amplification by iteration/post-processing, and shows fewer compute epochs than retraining on MNIST and CIFAR-10.
Problem
Machine unlearning with formal (epsilon, delta) guarantees for non-convex deep neural networks, where prior certified methods require convexity or smoothness and knowledge of smoothness constants, and retraining is costly.
Approach
Noisy fine-tuning on the retain data with gradient clipping or model clipping is treated as stochastic post-processing of the initial model, so privacy amplification by iteration yields (epsilon, delta)-unlearning without any loss-function assumptions; noise can be reduced by spreading it over many steps.
Key results
- On CIFAR-10, gradient and model clipping reach 40% accuracy in 10 epochs versus 18 for retraining from scratch and 12 for output perturbation.
- Table of epochs-to-target shows savings of up to 50% over retraining from scratch on CIFAR-10 (e.g., 30% accuracy in 3 epochs for model clipping vs 6 for retraining).
- With a ResNet-18 transfer setting, CIFAR-10 reaches 85% in 9 epochs, 10% faster than retraining and 47% faster than DP-SGD; CIFAR-100 reaches 60% in 32 epochs vs 34 for retraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gradient Clipping instead of retraining from scratch | Certified unlearning on CIFAR-10 · CIFAR-10, target accuracy 40% | epochs to target accuracy: 18 -> 10 | Helped |
| Use Model Clipping instead of retraining from scratch | Certified unlearning on CIFAR-10 · CIFAR-10, target accuracy 40% | epochs to target accuracy: 18 -> 10 | Helped |
| Use Gradient Clipping instead of Output Perturbation | Certified unlearning on CIFAR-10 · CIFAR-10, target accuracy 40% | epochs to target accuracy: 12 -> 10 | Helped |
| Use Model Clipping instead of Output Perturbation | Certified unlearning on CIFAR-10 · CIFAR-10, target accuracy 40% | epochs to target accuracy: 12 -> 10 | Helped |
| Use noisy fine-tuning instead of retraining from scratch | Certified unlearning with pretrained ResNet-18 feature extractor on CIFAR-10 · CIFAR-10, 85% accuracy | epochs to target accuracy: 10\,\% faster | Helped |
| Use noisy fine-tuning instead of DP-SGD with group privacy | Certified unlearning with pretrained ResNet-18 feature extractor on CIFAR-10 · CIFAR-10, 85% accuracy | epochs to target accuracy: 47\,\% faster | Helped |
| Use noisy fine-tuning instead of retraining from scratch | Certified unlearning with pretrained ResNet-18 feature extractor on CIFAR-100 · CIFAR-100, 60% accuracy | epochs to target accuracy: 32 versus 34 | Helped |
| Use noisy fine-tuning instead of DP-SGD with group privacy | Certified unlearning with pretrained ResNet-18 feature extractor on CIFAR-100 · CIFAR-100, within 50-epoch budget | accuracy: 60\,\% versus never exceeds 20\,\% | Helped |
Surprising findings
- Unlearning with aggressive clipping drives accuracy near zero, yet fine-tuning recovers quickly and surpasses retraining within about one epoch, suggesting retained useful information.
Limitations
- Curse of dimensionality inherent in differential privacy may limit scaling to models with very many parameters.
- Framework is designed for SGD in the unlearning stage only.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100 ImageNet
Benchmarks: MNIST CIFAR-10 CIFAR-100
Metrics: top-1 accuracy epochs to target accuracy compute budget
Baselines: retraining from scratch output perturbation DP-SGD with group privacy
Builds on: privacy amplification by iteration amplification by post-processing DP-SGD Rényi divergence shifted analysis
Closest papers by meaning
- The Utility and Complexity of In- and Out-of-Distribution Machine Unlearning (ICLR 2025 · similarity 0.93)
- Certified Machine Unlearning via Noisy Stochastic Gradient Descent (NeurIPS 2024 · similarity 0.92)
- Rewind-to-Delete: Certified Machine Unlearning for Nonconvex Functions (NeurIPS 2025 · similarity 0.92)
- Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearning (NeurIPS 2024 · similarity 0.92)
- A Certified Unlearning Approach without Access to Source Data (ICML 2025 · similarity 0.91)
- Towards Source-Free Machine Unlearning (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.