Research map / Optimization / Optimizers
Machine Unlearning under Overparameterization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a framework for machine unlearning in overparameterized models that defines the target as the minimum-complexity interpolator on the retain set and computes it via a first-order orthogonal-gradient relaxation (MinNorm-OG), with exact/approximate guarantees for linear models, linear networks, and 2-layer perceptrons.
Problem
Machine unlearning in the overparameterized regime, where many models interpolate the data, so loss-minimality over the retain set does not define a unique target and gradient-based unlearning methods receive vanishing gradients and fail.
Approach
Define the unlearning target as the minimum-complexity interpolator of the retain set and solve it by linearizing the interpolation constraints around the original model, restricting drift to be orthogonal to retain-set model gradients, plus a regularizer on drift size; the resulting algorithm alternates a norm-minimizing orthogonal projection step with retain-set loss descent.
Key results
- In Data Poisoning at 1000 epochs, MinNorm-OG reached median sup-norm distance 0.63 versus 1.17 for Retrain and 1400 for GA.
- In Representation Collapse with 0.1% retain data and 15 epochs, MinNorm-OG achieved 93.2% color-label accuracy versus 81.5% for the best baseline (NGP) (Retrain 96.3%).
- Per-epoch runtime of MinNorm-OG (0.78 s) is comparable to NGP (0.74 s) and below Retrain (0.90 s) on Tiny ImageNet with ResNet-50.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MinNorm-OG instead of the strongest listed baseline, Ridge | Data Poisoning, 1000 epochs · Data Poisoning, 1000 epochs | median sup-norm distance: 1.96 -> 0.63 | Helped |
| Use MinNorm-OG instead of Retrain | Data Poisoning, 100 epochs · Data Poisoning, 100 epochs | median sup-norm distance: 1.36 -> 1.08 | Helped |
| Use MinNorm-OG instead of Retrain | Data Poisoning, 10 epochs · Data Poisoning, 10 epochs | median sup-norm distance: 1.50 -> 1.50 | No clear effect |
| Use MinNorm-OG instead of NGP | Representation Collapse, retain 0.1%, 5 epochs · Representation Collapse, retain 0.1%, 5 epochs | accuracy (%): 45.9 -> 49.1 | Helped |
| Use MinNorm-OG instead of NGP | Representation Collapse, retain 0.1%, 10 epochs · Representation Collapse, retain 0.1%, 10 epochs | accuracy (%): 73.7 -> 78.7 | Helped |
| Use MinNorm-OG instead of NGP | Representation Collapse, retain 0.1%, 15 epochs · Representation Collapse, retain 0.1%, 15 epochs | accuracy (%): 81.5 -> 93.2 | Helped |
| Use MinNorm-OG instead of NGP | Representation Collapse, retain 1%, 5 epochs · Representation Collapse, retain 1%, 5 epochs | accuracy (%): 58.3 -> 59.5 | Helped |
| Use MinNorm-OG instead of NGP | Representation Collapse, retain 1%, 10 epochs · Representation Collapse, retain 1%, 10 epochs | accuracy (%): 92.4 -> 94.7 | Helped |
| Use MinNorm-OG instead of Retrain | Representation Collapse · Representation Collapse, all reported constraints | accuracy (%) | Hurt |
| Constrain perturbations to preserve orthogonality to retained-set model gradients | MinNorm-OG · Overparameterized interpolation | unlearning update | Helped |
| Add drift regularization to control perturbations (ablation) | Relaxed unlearning framework · Linear networks and two-layer perceptrons | exact unlearning feasibility | Helped |
| Restrict two-layer perceptron perturbations to the last layer (ablation) | Relaxed unlearning framework · Two-layer perceptron | active neurons | Helped |
Surprising findings
- Loss-gradient unlearning methods reduce to adding noise under overparameterization, since loss gradients vanish at interpolating solutions.
- Retrain performs worse than MinNorm-OG in the Data Poisoning experiment at 1000 epochs, as retraining from scratch with limited epochs does not fit the retain trend well.
Limitations
- Hmm, that closing tag was a mistake in my output; the function call is what matters, so I will resend it correctly below.
Experimental setup
Datasets: CIFAR-10 Tiny ImageNet synthetic sin(x) regression data
Benchmarks: Data Poisoning (sin(x) sup-norm distance) Multi-Class Label Erasure on CIFAR-10 and Tiny ImageNet Representation Collapse on CIFAR-10
Metrics: sup-norm distance to retain trend gray-test accuracy forget quality MSE to gray probability 1 color-label accuracy runtime per epoch
Baselines: Retrain GD GA NegGrad+ Noisy GD l1-Sparse Ridge SCRUB NPO SalUn
Builds on: influence functions NegGrad+ SCRUB SalUn NPO
Closest papers by meaning
- When to Forget? Complexity Trade-offs in Machine Unlearning (ICML 2025 · similarity 0.91)
- What makes unlearning hard and what to do about it (NeurIPS 2024 · similarity 0.90)
- Hessian-Free Online Certified Unlearning (ICLR 2025 · similarity 0.90)
- Data Unlearning in Diffusion Models (ICLR 2025 · similarity 0.90)
- The BatchNorm Illusion: Diagnosing Normalization Artifacts in Machine Unlearning Evaluation (NeurIPS 2026 · similarity 0.90)
- Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.