Research map / Adversarial robustness and security / Certified and empirical robustness
SAP: Corrective Machine Unlearning with Scaled Activation Projection for Label Noise Robustness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAP, a single-step SVD-based corrective unlearning method that selects low-loss trusted samples, estimates a clean activation subspace, and projects layer weights onto it to mitigate label noise.
Problem
Label corruption (synthetic or real-world mislabeling) degrades generalization of deep networks, and existing corrective unlearning methods need explicit identification of mislabeled vs. clean samples and/or costly retraining or hyperparameter search.
Approach
Select a small set of low cross-entropy-loss samples as trusted, compute SVD of their layer input activations, scale singular directions by a variance-based importance (scaling coefficient alpha), and fold the resulting projection matrix into the layer weights in a single update, without explicitly detecting mislabeled samples.
Key results
- On CIFAR10 with ResNet18 and 25% symmetric noise, SAP improves over Vanilla by about 6% (85.46 vs 79.47 average-style comparison reported in noise-robust table).
- SAP achieves average CIFAR10 accuracy of 87.66 vs 86.35 for the best other unlearning method (SCRUB with 1000/200 samples) across settings, and combined with noise-robust methods gives ~3.45% average gain on CIFAR10.
- On Clothing1M with ViT_B_16, SAP improves accuracy by 2.31% (69.12 to 71.43); average real-world gain is 1.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SAP after corrective unlearning on noisy training data | CIFAR10 models · CIFAR10 synthetic label noise (Table 1 average) | test accuracy: 86.35 -> 87.66 | Helped |
| Apply SAP after corrective unlearning on noisy training data | CIFAR100 models · CIFAR100 synthetic label noise (Table 1 average) | test accuracy: 62.79 -> 63.18 | Helped |
| Apply SAP after vanilla training | ResNet18 · CIFAR10 symmetric noise, η=0.25 | test accuracy: 79.47±0.46 -> 85.46±0.41; 5.99 | Helped |
| Apply SAP after noise-robust training | ResNet18 models · CIFAR10 symmetric noise, η=0.25 | test accuracy: +3.45 | Helped |
| Apply SAP after noise-robust training | ResNet18 models · CIFAR100 symmetric noise, η=0.25 | test accuracy: +2.98 | Helped |
| Apply SAP to the noisy-data model | InceptionResNetV2 · Mini-WebVision | test accuracy: 63.81±0.38 -> 64.73±0.53; 0.92 | Helped |
| Apply SAP to the noisy-data model | ResNet50 · Clothing1M | test accuracy: 67.48±0.64 -> 69.64±0.57; 2.16 | Helped |
| Apply SAP to the noisy-data model | ViT-B/16 · Clothing1M | test accuracy: 69.12±0.45 -> 71.43±0.60; 2.31 | Helped |
| Increase SCRUB retain and forget samples from 1000/200 to 5000/1000 (ablation) | SCRUB · CIFAR10 synthetic label noise (Table 1 average) | test accuracy: 86.35 -> 87.65 | Helped |
| Increase SCRUB retain and forget samples from 1000/200 to 5000/1000 (ablation) | SCRUB · CIFAR100 synthetic label noise (Table 1 average) | test accuracy: 62.79 -> 63.89 | Helped |
| Increase the number of trusted samples up to 1000 (ablation) | SAP · CIFAR100, η=25% | test accuracy | Helped |
| Increase the number of trusted samples beyond 1000 (ablation) | SAP · CIFAR100, η=25% | test accuracy | No clear effect |
Surprising findings
- Finetuning on low-loss samples was found to catastrophically decrease generalization in the authors' experiments, while SAP uses low-loss samples successfully.
- With a larger retain set (5000/1000), SCRUB matches or slightly beats SAP on CIFAR100 (by 0.71%).
Limitations
- Performance is sensitive to the scaling coefficient alpha, which must be tuned (alpha=30000 suggested for synthetic noise).
- SCRUB with larger retain/forget sets matches or exceeds SAP on some CIFAR100 settings.
- Gains on real-world noise (Mini-WebVision 0.92%, Clothing1M 2.16%) are smaller than on synthetic settings.
Experimental setup
Datasets: CIFAR10 CIFAR100 Mini-WebVision Clothing1M
Benchmarks: CIFAR10 CIFAR100 Mini-WebVision Clothing1M
Metrics: test accuracy
Baselines: Retrain Vanilla Finetune SSD SCRUB Logit Clip SAM MixUp MentorMix
Builds on: SSD SCRUB CF-k MentorMix
Closest papers by meaning
- Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise (NeurIPS 2018 · similarity 0.87)
- Distilling Effective Supervision From Severe Label Noise (CVPR 2020 · similarity 0.86)
- Mitigating Label Noise through Data Ambiguation (AAAI 2024 · similarity 0.85)
- Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic Optimization (NeurIPS 2025 · similarity 0.85)
- LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty (CVPR 2025 · similarity 0.85)
- Stable Unlearnable Example: Enhancing the Robustness of Unlearnable Examples via Stable Error-Minimizing Noise (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.