Research map / Optimization / Implicit bias and generalization
Ascent Fails to Forget
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows gradient ascent/descent-ascent unlearning fails in practice and theoretically because forget and retain sets are statistically dependent, with logistic regression and toy examples showing divergence from the oracle.
Problem
Gradient ascent-based (descent-ascent) machine unlearning is widely used but unreliable; the paper asks whether statistical dependence between forget and retain sets causes these methods to degrade models instead of approximating retraining.
Approach
Attributes failure of descent-ascent unlearning to inter-set statistical dependence (even simple correlations): shows random forget sets make forget-set metrics track test metrics, and in logistic regression and a low-dimensional sigmoidal toy the DA solution lies on the opposite side of the pretrained solution from the oracle, and can be trapped in inferior local minima.
Key results
- On CIFAR-10 with ResNet-9, gradient ascent and gradient descent/ascent either fail to move from the pretrained model or severely degrade performance, with no clear stopping criterion.
- Theory shows for 1D logistic regression that the DA solution lies on the opposite side of the pretrained solution from the oracle (sign of (w_DA - w_D)(w_D - w_R) >= 0 for alpha <= |F|/|R|).
- In a 2D toy example with 4 weighted points, GDA converges to a solution with 10 correct and 2 incorrect, while retraining keeps the pretrained boundary (11 correct, 1 incorrect).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply descent-ascent unlearning to correlated forget and retain sets | Gradient Descent/Ascent unlearning · Correlated forget and retain data | distance from oracle solution | Hurt |
| Use descent-ascent on forget sets aligned with top principal components | Gradient Descent/Ascent unlearning · Structurally correlated forget sets | model performance | Hurt |
| Use ascent to degrade metrics on a random forget set | Gradient Ascent unlearning · Random forget sets | overall test performance | Hurt |
| Perform GDA instead of retraining on the retain set | Sigmoidal network toy example · Correlated 2D classification toy | correctly classified samples: 11 correct, 1 incorrect -> 10 correct, 2 incorrect | Hurt |
| Use GDA to find a decision boundary that classifies all samples | Sigmoidal network toy example · Correlated 2D classification toy | correctly classified samples: 10 correct, 0 incorrect; actual algorithm solution: 10 correct, 2 incorrect | Mixed |
| Select hyperparameters after inspecting the final target metric | Gradient Ascent unlearning · Small random forget sets | apparent unlearning success | Helped |
Surprising findings
- Random forget sets make forget-set accuracy statistically indistinguishable from test accuracy, so degrading forget metrics inevitably harms test performance.
- DA solutions can end up farther from the oracle than the original pretrained model, and the failure can be inescapable via finetuning on the retain set.
Limitations
- Theory is restricted to logistic regression with exponential loss and a toy sigmoidal network; extending to multi-layer networks is left open.
- Quantifying which correlations are most harmful remains open.
Experimental setup
Datasets: CIFAR-10 ImageNetLiving-17 MNIST FashionMNIST
Benchmarks: CIFAR-10 ImageNetLiving-17 MNIST FashionMNIST
Metrics: KLoM accuracy KL divergence of margins
Baselines: Gradient Ascent Gradient Descent/Ascent retraining (oracle)
Builds on: KLoM certified unlearning SCRUB
Closest papers by meaning
- When to Forget? Complexity Trade-offs in Machine Unlearning (ICML 2025 · similarity 0.89)
- From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularization (NeurIPS 2025 · similarity 0.88)
- What makes unlearning hard and what to do about it (NeurIPS 2024 · similarity 0.88)
- RULE: Reinforcement UnLEarning Achieves Forget-retain Pareto Optimality (NeurIPS 2025 · similarity 0.88)
- Fast Machine Unlearning without Retraining through Selective Synaptic Dampening (AAAI 2024 · similarity 0.87)
- Machine Unlearning under Overparameterization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.