Research map / Image generation / Autoregressive and latent generation
DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DualOptim, which optimizes the forget loss with Adam and the retain loss with the pretraining optimizer using decoupled momentum, to make approximate machine unlearning more effective and stable across classification, diffusion generation, and LLM unlearning.
Problem
Approximate machine unlearning methods are highly sensitive to hyperparameters and show unstable or suboptimal forgetting/utility trade-offs across forget sets, tasks, and models.
Approach
Use two separate optimizers for the forget and retain objectives in alternating unlearning: Adam (adaptive learning rate) for the forget loss and the pretraining optimizer for the retain loss, so each objective keeps its own momentum; a variance analysis argues this reduces parameter variance versus a shared momentum.
Key results
- On 10% CIFAR-10 random-subset unlearning with ResNet-18, SFRon+DualOptim reduces average std from 2.33% to 0.66% and gap from 0.51% to 0.44%.
- On TOFU with LLaMA 2 (forget 10%), ME+GD+DO reaches average score 0.8470 versus 0.8423 for ME+GD, and IDK+AP+DO reaches 0.7609 versus 0.7452.
- DualOptim with SFRon on class-wise CIFAR-10 DDPM lowers mean FID from 19.71 to 18.77 while reducing FA mean from 1.52% to 0.24%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DualOptim to SFRon | SFRon on CIFAR-10 random subset 10% · CIFAR-10 random subset 10% | Gap ↓: 0.51 -> 0.44 | Helped |
| Add DualOptim to SFRon | SFRon on CIFAR-10 random subset 10% · CIFAR-10 random subset 10% | Std ↓: 2.33 -> 0.66 | Helped |
| Add DualOptim to SCRUB | SCRUB on CIFAR-10 random subset 10% · CIFAR-10 random subset 10% | Gap ↓: 2.33 -> 0.88 | Helped |
| Add DualOptim to SalUn | SalUn on CIFAR-10 random subset 10% · CIFAR-10 random subset 10% | Gap ↓: 3.47 -> 0.93 | Helped |
| Add DualOptim to SFRon | SFRon on TinyImageNet random subset 10% · TinyImageNet random subset 10% | Gap ↓: 3.73 -> 2.68 | Helped |
| Add DualOptim to SFRon | SFRon on CIFAR-10 class-wise unlearning · CIFAR-10 class-wise unlearning | FID ↓: 19.71 -> 18.77 | Helped |
| Add DualOptim to SFRon | SFRon on CIFAR-10 class-wise unlearning · CIFAR-10 class-wise unlearning | FA ↓: 1.52 -> 0.24 | Helped |
| Add DualOptim to ME+GD | ME+GD on Phi-1.5 · TOFU forget 1% | Avg.: 0.4441 -> 0.5889 | Helped |
| Add DualOptim to IDK+AP | IDK+AP on Phi-1.5 · TOFU forget 1% | Avg.: 0.5063 -> 0.5629 | Helped |
| Use Adam for forgetting and SGD for retaining rather than one shared optimizer (ablation) | SFRon on CIFAR-10 random subset 10% · CIFAR-10 random subset 10% | Gap ↓: 0.51 -> 0.44 | Helped |
| Replace SGD with Adam for retaining (ablation) | SFRon with Adam for forgetting on CIFAR-10 random subset 10% · CIFAR-10 random subset 10% | Gap ↓: 0.44 -> 0.55 | Hurt |
| Use SGD for forgetting and Adam for retaining instead of Adam for forgetting and SGD for retaining (ablation) | SFRon on CIFAR-10 random subset 10% · CIFAR-10 random subset 10% | Gap ↓: 0.44 -> 1.73 | Hurt |
Surprising findings
- A shared SGD optimizer gives competitive average performance but high instability, while Lion gives low variability but weaker unlearning.
- Decoupled momentum adds about 1.5x running time and roughly doubles optimizer-state memory for full-parameter LLM tuning.
Limitations
- Decoupled momentum doubles optimizer-state memory for full-parameter fine-tuning.
- Theoretical analysis relies on assumptions such as bounded correlations and Lipschitz smoothness, with near-zero cross-correlation between forget and retain gradients assumed.
- Hyperparameters for DualOptim are themselves tuned per setting, so sensitivity is reduced but not eliminated.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 SVHN TinyImageNet ImageNet TOFU
Benchmarks: CIFAR-10 TinyImageNet CIFAR-100 SVHN ImageNet class-wise TOFU
Metrics: forget accuracy retain accuracy test accuracy membership inference attack success average gap to retraining standard deviation FID model capability forget efficacy
Baselines: Fine-tune (FT) Gradient Ascent (GA) Random Label (RL) SCRUB SalUn SFRon SA GA+GD NPO+GD ME+GD DPO+GD IDK+AP
Builds on: SFRon SalUn SCRUB Adam TOFU benchmark
Closest papers by meaning
- LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty (CVPR 2025 · similarity 0.91)
- When to Forget? Complexity Trade-offs in Machine Unlearning (ICML 2025 · similarity 0.89)
- The BatchNorm Illusion: Diagnosing Normalization Artifacts in Machine Unlearning Evaluation (NeurIPS 2026 · similarity 0.89)
- Machine Unlearning for Image-to-Image Generative Models (ICLR 2024 · similarity 0.88)
- Machine Unlearning under Overparameterization (NeurIPS 2025 · similarity 0.88)
- Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.