Research map / Continual learning / Continual fine-tuning of LLMs
Distribution-Level Feature Distancing for Machine Unlearning: Towards a Better Trade-off Between Model Utility and Forgetting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Distribution-Level Feature Distancing (DLFD), an optimal-transport-based data-synthesis method for instance-level machine unlearning on facial datasets that aims to avoid correlation collapse and preserve task utility.
Problem
Machine unlearning of specific face images often causes utility loss ("correlation collapse") because error-maximizing methods weaken feature-label correlations; the paper targets forgetting with task-agnostic utility preservation.
Approach
Synthesize perturbed retain samples by maximizing an optimal-transport (Sinkhorn) distance between retain-feature and forget-feature distributions, with a classification loss weighted by a linearly increasing factor and a dynamic stop-forgetting rule, so forgetting is achieved within one epoch while task features are kept.
Key results
- On facial age (MUFAC, ResNet18), DLFD reaches NoMUS 0.7698 vs 0.7088 for UNSIR and forgetting score 0.0385 vs 0.0887 for UNSIR.
- On facial emotion (RAF-DB, ResNet18), DLFD achieves NoMUS 0.7934 but test accuracy drops to 0.6613 vs 0.7535 original.
- On facial gender, DLFD reports NoMUS 0.9192 vs 0.9119 for fine-tuning and forgetting score 0.0306.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add classification loss preservation to feature distribution optimization (ablation) | DLFD · Age | NoMUS: 0.7021 -> 0.7536 | Helped |
| Add classification loss preservation to feature distribution optimization (ablation) | DLFD · Emotion | NoMUS: 0.7199 -> 0.7536 | Helped |
| Add classification loss preservation to feature distribution optimization (ablation) | DLFD · Multi-Attributes | NoMUS: 0.9423 -> 0.9420 | Hurt |
| Add classification loss preservation to feature distribution optimization (ablation) | DLFD · Gender | NoMUS: 0.8898 -> 0.9170 | Helped |
| Add dynamic forgetting to DLFD with classification loss preservation (ablation) | DLFD · Age | NoMUS: 0.7536 -> 0.7698 | Helped |
| Add dynamic forgetting to DLFD with classification loss preservation (ablation) | DLFD · Emotion | NoMUS: 0.7536 -> 0.7934 | Helped |
| Add dynamic forgetting to DLFD with classification loss preservation (ablation) | DLFD · Multi-Attributes | NoMUS: 0.9420 -> 0.9433 | Helped |
| Add dynamic forgetting to DLFD with classification loss preservation (ablation) | DLFD · Gender | NoMUS: 0.9170 -> 0.9192 | Helped |
Surprising findings
- On the emotion task DLFD's test accuracy (0.6613) falls well below the original model (0.7535), so the NoMUS gain comes largely from the forgetting term.
- Naive error-maximizing unlearning can push forget-sample loss above unseen-sample loss when the forget set is small (<100), leaking membership.
Limitations
- Performance may degrade when retain and forget feature distributions overlap heavily.
- MIA-based forgetting score may not reflect true unlearning when models are very well trained.
- Emotion task shows a substantial test accuracy drop relative to the original model.
Experimental setup
Datasets: MUFAC RAF-DB MUCAC CelebA
Benchmarks: MUFAC RAF-DB MUCAC
Metrics: test accuracy forgetting score (MIA-based) NoMUS
Baselines: Fine-tuning NegGrad CF-k EU-k UNSIR BadTeaching SCRUB Retrained
Builds on: UNSIR SCRUB BadTeaching Fisher Forgetting
Closest papers by meaning
- Learning to Unlearn: Instance-Wise Unlearning for Pre-trained Classifiers (AAAI 2024 · similarity 0.88)
- Not All Wrong is Bad: Using Adversarial Examples for Unlearning (ICML 2025 · similarity 0.88)
- The Unseen Threat: Residual Knowledge in Machine Unlearning under Perturbed Samples (NeurIPS 2025 · similarity 0.88)
- Decoupled Distillation to Erase: A General Unlearning Method for Any Class-centric Tasks (CVPR 2025 · similarity 0.88)
- Label-Agnostic Forgetting: A Supervision-Free Unlearning in Deep Models (ICLR 2024 · similarity 0.88)
- The Utility and Complexity of In- and Out-of-Distribution Machine Unlearning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.