Research map / Label-efficient and robust learning / Recognition and long-tail

Severing Spurious Correlations with Data Pruning

ICLR 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Data augmentation and generalization bias

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that spurious correlations are learned mainly from a small set of hard-core-feature samples containing the spurious feature, and proposes a training-free data pruning method that removes the hardest samples per class to sever spurious correlations in settings where spurious features are unidentifiable.

Problem

Existing spurious-correlation mitigation methods rely on identifying which training samples contain spurious features, which becomes impossible when the spurious signal is weaker than the core signal; the paper asks how to overcome spurious correlations in such settings.

Approach

Spurious correlations are primarily learned from the few samples that contain both the spurious feature and hard-to-learn core features; pruning the hardest fraction of training samples per class (based on early-training error) severs these correlations without needing spurious-feature labels, group labels, or hyperparameter tuning in unidentifiable settings.

Key results

• On Waterbirds, the pruning method reaches 90.93% worst-group accuracy (mean 92.48%), exceeding the listed baselines including DFR^Tr (90.2%) and PDE (90.3%).
• On MultiNLI, worst-group accuracy reaches 75.88%, below gDRO's 77.7% but above JTT (72.6%) and DFR^Tr (71.5%).
• In the CIFAR-10S experiment, injecting the spurious feature into the 100 hardest samples yields 58.0% spurious misclassifications in the identifiable setting and 13.76% in the unidentifiable setting.

Measured effects reported in the paper

Surprising findings

• Spurious feature reliance grows super-linearly with the difficulty of the core features of the 100 samples carrying the spurious feature.
• Excluding a small fraction (about 1% of total train set) of hard samples containing the spurious feature improves worst-group accuracy in CelebA, while excluding up to 97% of easy spurious samples does not.

Limitations

• Pruning hardest samples per class is only shown to work in unidentifiable settings with a particular difficulty ranking; in identifiable settings it prunes the wrong samples, so group labels are needed there.
• Results on MultiNLI are below gDRO, and Waterbirds mean accuracy is lower than ERM (92.48 vs 98.10).

Experimental setup

Datasets: CIFAR-10 CelebA Hard ImageNet Waterbirds MultiNLI

Benchmarks: CIFAR-10S CelebA Hard ImageNet Waterbirds MultiNLI

Metrics: worst-group accuracy mean accuracy spurious misclassifications

Baselines: ERM CnC JTT gDRO DFR PDE

Builds on: Just Train Twice (JTT) Correct-N-Contrast (CnC) Deep Feature Reweighting (DFR) Group DRO Prediction Depth / data pruning by error scores

Closest papers by meaning

• Let Samples Speak: Mitigating Spurious Correlation by Exploiting the Clusterness of Samples (CVPR 2025 · similarity 0.91)
• Complexity Matters: Feature Learning in the Presence of Spurious Correlations (ICML 2024 · similarity 0.88)
• How Spurious Features are Memorized: Precise Analysis for Random and NTK Features (ICML 2024 · similarity 0.86)
• Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization (ICML 2025 · similarity 0.85)
• Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.85)
• NeuronTune: Towards Self-Guided Spurious Bias Mitigation (ICML 2025 · similarity 0.85)