Research map / Label-efficient and robust learning / Noisy labels
Enhancing Robustness of Last Layer Two-Stage Fair Model Corrections
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes kNN label spreading as a drop-in preprocessing step that corrects symmetric label noise before two-stage last-layer retraining (RAD/SELF), improving worst-group accuracy under noisy labels.
Problem
Two-stage last-layer retraining methods for worst-group accuracy (without group labels) collapse under symmetric label noise because the error set used for retraining fills with noisy majority points.
Approach
Apply iterative kNN majority-vote label spreading on the latent embeddings of the base model to clean noisy labels, then pass cleaned labels to any two-stage last-layer method (kNN-RAD, kNN-SELF), exploiting the near-linear separability and clustering of clean embeddings.
Key results
- On CelebA at 30% symmetric label noise, kNN-RAD reaches 79.5 WGA versus 0 for vanilla RAD and 77.8 for END.
- On Waterbirds at 20% noise, kNN-RAD achieves 89.93 WGA versus 50.74 for RAD and 83.2 for END.
- kNN-RAD matches domain-aware GDS and GUW on CivilComments at up to 30% noise (78.56 vs 81.21 for GDS at 30%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add kNN label spreading before RAD (ablation) | RAD · CelebA at 20% and 30% label noise | WGA: 0 (0) -> 82.29 (1.25); 0 (0) -> 79.5 (1.91) | Helped |
| Add kNN label spreading before RAD (ablation) | RAD · Waterbirds at 10%, 20%, and 30% label noise | WGA: 79.33 (1.38) -> 90.72 (0.63); 50.74 (2.23) -> 89.93 (1.10); 19.52 (1.91) -> 86.90 (3.07) | Helped |
| Add kNN label spreading before RAD (ablation) | RAD · CivilComments at 10%, 20%, and 30% label noise | WGA: 79.25 (0.44) -> 81.15 (0.60); 77.45 (0.71) -> 80.70 (0.62); 54.36 (1.30) -> 78.56 (1.52) | Helped |
| Add kNN label spreading before RAD (ablation) | RAD · CMNIST at 10%, 20%, and 30% label noise | WGA: 89.62 (0.72) -> 92.46 (0.92); 89.52 (0.65) -> 91.95 (0.86); 88.78 (0.89) -> 90.50 (3.59) | Helped |
| Add kNN label spreading before SELF (ablation) | SELF · Waterbirds at 10%, 20%, and 30% label noise | WGA: 7.58 (2.77) -> 89.55 (0.65); 1.38 (0.20) -> 88.24 (2.93); 0.66 (0.14) -> 82.44 (10.43) | Helped |
| Add kNN label spreading before SELF (ablation) | SELF · CelebA at 10%, 20%, and 30% label noise | WGA: 60.48 (6.09) -> 72.96 (22.96); 59.99 (4.52) -> 75.77 (8.02); 60.71 (3.70) -> 73.98 (4.40) | Helped |
| Add kNN label spreading before SELF (ablation) | SELF · CMNIST at 10%, 20%, and 30% label noise | WGA: 89.98 (1.16) -> 92.81 (0.88); 88.05 (2.35) -> 91.06 (1.06); 90.58 (1.12) -> 90.16 (3.67) | Mixed |
| Train the base model using 20% symmetric label noise (ablation) | RAD and SELF corrections · CelebA, noisy base-model embeddings | WGA: RAD 83.89 (0) -> 0 (0); SELF 83.48 (0) -> 59.99 (4.52) | Hurt |
| Use kNN label spreading with noisy base-model embeddings (ablation) | RAD · CelebA, 20% noise in retraining labels | WGA: 0 (0) -> 76.56 (2.24) | Helped |
| Use kNN label spreading with noisy base-model embeddings (ablation) | RAD · Waterbirds, 20% noise in retraining labels | WGA: 32.26 (2.99) -> 73.78 (3.29) | Helped |
| Increase the number of label-spreading rounds or neighbors too far (ablation) | kNN label spreading · CMNIST label correction | Clean accuracy | Hurt |
Surprising findings
- Vanilla RAD and SELF collapse to 0 or near-0 worst-group accuracy under moderate label noise, even though they perform well at 0% noise.
- Using a robust loss (alpha-loss) in the identification step does not prevent failure, since robust models still misclassify noisy points and place them in the error set.
Limitations
- Assumes embeddings come from a model trained on clean data; robustness to noisy embeddings is only partially verified.
- Number of neighbors k is a hyperparameter requiring tuning, and CMNIST needs smaller k due to non-collapsed clusters.
- Evaluated only on symmetric label noise and four datasets.
Experimental setup
Datasets: CMNIST CelebA Waterbirds CivilComments
Benchmarks: CMNIST CelebA Waterbirds CivilComments
Metrics: worst-group accuracy
Baselines: RAD SELF END GUW GDS
Builds on: RAD SELF END Gao et al. kNN label noise theory deep feature reweighting
Closest papers by meaning
- Towards Last-layer Retraining for Group Robustness with Fewer Annotations (NeurIPS 2023 · similarity 0.89)
- Retraining with Predicted Hard Labels Provably Increases Model Accuracy (ICML 2025 · similarity 0.88)
- Group-robust Sample Reweighting for Subpopulation Shifts via Influence Functions (ICLR 2025 · similarity 0.86)
- Regroup Median Loss for Combating Label Noise (AAAI 2024 · similarity 0.86)
- The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlations (NeurIPS 2024 · similarity 0.85)
- Mitigating Label Bias in Machine Learning: Fairness through Confident Learning (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.