Research map / Optimization / Optimizers

Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement

NeurIPS 2024 · Optimization › Optimizers · approach: Surrogate losses and robust solvers · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives a steepest-descent view of approximate machine unlearning under a remain-preserving KL manifold and proposes SFR-on, a fast-slow weight method that implicitly approximates Hessian-modulated forgetting updates for classification and diffusion-model unlearning.

Problem

Approximate machine unlearning via gradient methods uses Euclidean geometry, which treats all parameters equally and can harm retained performance; exact Hessian-based manifold updates are intractable for large models.

Approach

Decomposes the steepest-descent unlearning direction into weighted forgetting gradient ascent, remaining gradient descent, and a weight saliency matrix; embeds updates in a KL-on-remaining-output manifold; and approximates the inverse remaining Hessian implicitly via a fast-slow weight update (inner fine-tuning on remaining data, outer interpolation), plus adaptive sample-wise forgetting coefficie

Key results

• On 10% random subset forgetting of CIFAR-10 with ResNet-18, SFR-on achieves Avg.D 1.20 and D_KL 0.15, lower than all baselines (e.g., SalUn Avg.D 3.03, D_KL 0.27).
• Class-wise forgetting on ImageNet with DiT: SFR-on reaches 0.00 FA on all five classes in 500 steps versus 10000 steps for SA and SalUn.
• On Stable Diffusion nudity erasure over 9406 I2P prompts, SFR-on yields 3 exposed body parts total versus 46 for SalUn and 1799 for SD V1.4.

Measured effects reported in the paper

Surprising findings

• Plain joint fine-tuning plus gradient ascent (R baseline) is reported to yield distorted remaining-class images in generation, which the Hessian-approximated update largely fixes.
• The fast-slow update is said to need no task-specific loss changes, using CE or MSE for the forgetting term.

Limitations

• Computing the sample-wise coefficients relies on loss magnitude as a heuristic, which the authors note is limited and potentially biased.
• For large generative models, the remaining fine-tuning and Fisher estimation use an equal-size random remaining subset rather than the full remaining set.
• RT for DiT is infeasible, so the reference is a simulated RT† rather than true retraining.

Experimental setup

Datasets: CIFAR-10 TinyImageNet CIFAR-100 SVHN ImageNet LAION I2P

Benchmarks: CIFAR-10 random subset forgetting TinyImageNet random subset forgetting CIFAR-10 class-wise forgetting with DDPM ImageNet class-wise forgetting with DiT nudity concept erasure in Stable Diffusion with I2P prompts

Metrics: FA RA TA MIA Avg.D KL divergence to retrain RTE FID NudeNet exposed body part counts

Baselines: FT GA RL BT SalUn SA SCRUB ESD

Builds on: Selective Amnesia SalUn Scrub / bad teacher unlearning fast-slow weights Selective Synaptic Dampening

Closest papers by meaning

• Machine Unlearning under Overparameterization (NeurIPS 2025 · similarity 0.89)
• LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty (CVPR 2025 · similarity 0.89)
• Hessian-Free Online Certified Unlearning (ICLR 2025 · similarity 0.89)
• The BatchNorm Illusion: Diagnosing Normalization Artifacts in Machine Unlearning Evaluation (NeurIPS 2026 · similarity 0.88)
• Machine Unlearning in Low-Dimensional Feature Subspace (NeurIPS 2026 · similarity 0.88)
• Certified Machine Unlearning via Noisy Stochastic Gradient Descent (NeurIPS 2024 · similarity 0.88)