Research map / Label-efficient and robust learning / Recognition and long-tail
LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoTUS, a machine unlearning method that smooths teacher logits with a dynamically scheduled, accuracy-driven Gumbel-Softmax temperature to remove forget-set influence without retraining, and introduces the retrain-free RF-JSD metric.
Problem
Approximate machine unlearning of training samples from pre-trained models, especially large-scale datasets like ImageNet1k where retraining from scratch is impractical and training data access may be limited.
Approach
Treat unlearning as output-space entropy control: a knowledge-distillation student learns from a teacher whose logits are smoothed via Gumbel-Softmax with a temperature adjusted by the accuracy gap between forget samples (student) and unseen data (teacher), motivated by an information-theoretic (Fano-based) accuracy objective.
Key results
- On ImageNet1k with ViT, LoTUS achieves RF-JSD 1.11e-4 versus 2.22e-4 for Finetuning and 1.24e-4 for SCRUB, with 10.72 min runtime versus 16.24 min for Finetuning.
- On TinyImageNet with ViT (10% forget), LoTUS reaches Avg Gap 0.0150 versus 0.0225 for the best baselines and 0.0925 for SalUn.
- RF-JSD correlates with JSD with mean Pearson coefficient 0.92±0.04 (p=0.001).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply LoTUS instead of the strongest listed baseline, Finetuning | ViT · TinyImageNet, 10% forget | Avg Gap: 0.0175 -> 0.0150 | Helped |
| Apply LoTUS instead of the strongest listed baseline, RndLbl | ViT · CIFAR-100, 10% forget | Avg Gap: 0.0175 -> 0.0125 | Helped |
| Apply LoTUS instead of the strongest listed baseline, RndLbl | ViT · MUFAC, 10% forget | Avg Gap: 0.0200 -> 0.0200 | No clear effect |
| Apply LoTUS instead of Finetuning | ResNet18 · TinyImageNet, 10% forget | Avg Gap: 0.2200 -> 0.1675 | Helped |
| Apply LoTUS instead of SCRUB | ResNet18 · CIFAR-100, 10% forget | Avg Gap: 0.1650 -> 0.1200 | Helped |
| Apply LoTUS instead of BadT | ResNet18 · MUFAC, 10% forget | Avg Gap: 0.1025 -> 0.1250 | Hurt |
| Use synthetic CIFAKE data rather than the real unseen set | LoTUS on ViT · CIFAR-10, 10% forget | Avg Gap: 0.0050 -> 0.0075 | Hurt |
| Use synthetic CIFAKE data rather than the real unseen set | LoTUS on ResNet18 · CIFAR-10, 10% forget | Avg Gap: 0.0350 -> 0.0300 | Helped |
| Apply LoTUS instead of SCRUB | ViT · CIFAR-100, 50% forget | Avg Gap: 0.0350 -> 0.0225 | Helped |
| Apply LoTUS instead of SalUn | ViT · CIFAR-10, 50% forget | Avg Gap: 0.0050 -> 0.0050 | No clear effect |
| Apply LoTUS instead of SCRUB | ResNet18 · CIFAR-100, 50% forget | Avg Gap: 0.2225 -> 0.1725 | Helped |
| Apply LoTUS instead of BadT | ResNet18 · CIFAR-10, 50% forget | Avg Gap: 0.0625 -> 0.0650 | Hurt |
Surprising findings
- The established Avg Gap metric can produce misleading results, with cases where it fails to detect unlearning misconceptions that JSD captures.
- LoTUS is not the best on MUFAC with ResNet18 by Avg Gap, where it is second-best.
Limitations
- Experiments on class unlearning are less extensive than on instance-wise unlearning.
- Performance on MUFAC with ResNet18 is second-best rather than best by Avg Gap.
- Theoretical assumptions (distributional similarity of forget and unseen sets) are relaxed only empirically.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 MUFAC TinyImageNet CIFAKE ImageNet1k
Benchmarks: CIFAR-10 CIFAR-100 MUFAC TinyImageNet ImageNet1k
Metrics: Avg Gap JSD RF-JSD Time (min) MIA accuracy forget/retain/test accuracy
Baselines: Finetuning NegGrad+ Random Labeling (RndLbl) Bad Teacher (BadT) SCRUB SSD UNSIR SalUn
Builds on: Bad Teacher SCRUB SalUn Random Labeling JSD evaluation of unlearning
Closest papers by meaning
- When to Forget? Complexity Trade-offs in Machine Unlearning (ICML 2025 · similarity 0.92)
- DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers (NeurIPS 2025 · similarity 0.91)
- Machine Unlearning for Image-to-Image Generative Models (ICLR 2024 · similarity 0.91)
- The BatchNorm Illusion: Diagnosing Normalization Artifacts in Machine Unlearning Evaluation (NeurIPS 2026 · similarity 0.91)
- Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models (NeurIPS 2024 · similarity 0.90)
- Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.