Research map / Continual learning / Online and replay-based

Forgetting Has Neighbors: Localized Collateral Forgetting in Machine Unlearning

NeurIPS 2026 · Continual learning › Online and replay-based · approach: Theoretical analysis of forgetting · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that gradient-ascent and random-label unlearning leave example-level discrepancies from retraining that concentrate near the forget set, and proposes Local Teacher Distillation, which replaces random forget-set targets with soft labels from a small teacher trained on nearby retained examples.

Problem

Aggregate unlearning metrics hide localized failures: unlearned models deviate from retraining on retained and test examples close to the forget set in representation space.

Approach

Identifies target mismatch (random forget-set targets inconsistent with retraining's local prediction structure) as the mechanism of localized collateral forgetting, and replaces random labels with soft labels from a small ResNet-8 teacher trained on the k retained examples most similar to the average forget embedding.

Key results

• On CIFAR-100 with 50% couch deletion, LTD achieves the lowest Avg. Gap (2.4) versus 6.6 for the next best method (IU) and 21.2 for FT.
• LTD's affected-class RA is 97.4% versus 66.6% for GA and 74.6% for IU, showing much smaller affected-class degradation.
• Nearest-neighbor support yields Avg. affected-class Avg. Gap 4.9 vs 12.6-24.3 for baselines, and random or farthest support fails to match retraining's forget-class behavior.

Surprising findings

• LTD's UA (53.4%) is further from retrain (47.4%) than some baselines in absolute deviation terms on the forget set, yet it achieves the lowest Avg. Gap, indicating that matching retraining does not require minimizing forget-set accuracy.
• Retain fine-tuning alone (RL+FT) leaves residual discrepancy on query points in the forget span even when it restores agreement on retained training points.

Limitations

• Relies on the quality of the local support set and representation used for selection; heterogeneous forget sets may need clustering or multiple teachers.
• Evaluated only on controlled image-classification partial-deletion settings (CIFAR-100, SVHN), not larger architectures or deletion regimes.
• Theoretical analysis is for linear and logistic regression settings and stylized assumptions.

Experimental setup

Datasets: CIFAR-100 SVHN

Benchmarks: CIFAR-100 partial-class deletion

Metrics: UA RA TA MIA Avg. Gap RTE affected-class accuracy similarity-binned accuracy drop vs retrain

Baselines: RL FT GA SalUn IU AMUN

Builds on: gradient ascent unlearning random labeling unlearning SalUn Influence Unlearning AMUN

Closest papers by meaning

• Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance (NeurIPS 2026 · similarity 0.91)
• When to Forget? Complexity Trade-offs in Machine Unlearning (ICML 2025 · similarity 0.90)
• Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models (AAAI 2025 · similarity 0.89)
• RA$^2$: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning (NeurIPS 2026 · similarity 0.89)
• From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularization (NeurIPS 2025 · similarity 0.89)
• Fast Machine Unlearning without Retraining through Selective Synaptic Dampening (AAAI 2024 · similarity 0.89)