Research map / Label-efficient and robust learning / Recognition and long-tail
Decompose-and-Compose: A Compositional Approach to Mitigating Spurious Correlation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Decompose-and-Compose (DaC), which uses ERM attribution maps and adaptive masking to split low-loss images into causal and non-causal parts, then recombines them across labels to upweight minority groups and retrain the last layer without group labels.
Problem
Spurious correlation between non-causal image components and labels causes ERM models to fail on minority groups under correlation shift; existing group-balancing methods need group labels, while label-free methods upweight high-loss samples that may be hard rather than minority.
Approach
Use xGradCAM attributions and an elbow-based adaptive masking of a pretrained ERM model to decompose images into parts, then combine the predictive part of one image with the non-predictive part of an image from a different class to synthesize minority-group samples, retraining only the last layer with a combined loss.
Key results
- DaC achieves 92.3% worst-group accuracy on Waterbirds and 78.3% on Metashift, outperforming label-free baselines and matching DFR on these datasets.
- On Dominoes, DaC reaches 89.2% worst-group accuracy versus 72.8% for ERM base.
- On CelebA, DaC's worst-group accuracy (81.9%) lags behind DFR (88.3%) and Group DRO (88.9%), which use group labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add adaptive-mask image combining and retrain the last layer (DaC) | ERM ResNet-50 · Waterbirds | worst group accuracy: 70.8±0.5 -> 92.3±0.4 | Helped |
| add adaptive-mask image combining and retrain the last layer (DaC) | ERM ResNet-50 · Waterbirds | average group accuracy: 91.6±0.1 -> 95.3±0.4 | Helped |
| add adaptive-mask image combining and retrain the last layer (DaC) | ERM ResNet-50 · CelebA | worst group accuracy: 41.7 -> 81.9±0.7 | Helped |
| add adaptive-mask image combining and retrain the last layer (DaC) | ERM ResNet-50 · CelebA | average group accuracy: 96.0 -> 91.4±1.1 | Hurt |
| add adaptive-mask image combining and retrain the last layer (DaC) | ERM ResNet-50 · Metashift | worst group accuracy: 61.3±3.4 -> 78.3±1.6 | Helped |
| add adaptive-mask image combining and retrain the last layer (DaC) | ERM ResNet-50 · Metashift | average group accuracy: 73.9±1.5 -> 79.3±0.1 | Helped |
| add adaptive-mask image combining and retrain the last layer (DaC) | ERM ResNet-50 · Dominoes | worst group accuracy: 72.8±1.6 -> 89.2±0.1 | Helped |
| add adaptive-mask image combining and retrain the last layer (DaC) | ERM ResNet-50 · Dominoes | average group accuracy: 88.5±0.3 -> 92.2±0.3 | Helped |
| use correctly classified samples for combination instead of tuning selected-sample proportion q (ablation) | DaC · CelebA | worst group accuracy: 81.9±0.7 -> 76.11±0 | Hurt |
| use correctly classified samples for combination instead of tuning selected-sample proportion q (ablation) | DaC · Metashift | average group accuracy: 79.3±0.1 -> 80.0±1.4 | Helped |
| increase the number of selected samples used for combining (ablation) | DaC · multiple datasets | worst group accuracy | Hurt |
| set combination-loss weight α to at least 1 (ablation) | DaC · multiple datasets | worst group accuracy | Helped |
Surprising findings
- ERM does not always attend to the spurious parts; attention depends on how predictive causal vs non-causal parts are across the dataset, and on Waterbirds ERM attends more to the foreground.
- The method does not use the CelebA spurious structure it was designed for, and underperforms group-label methods there.
Limitations
- Designed for part-level spurious correlations; does not address spurious attributes such as colour, and underperforms on CelebA.
- Hyperparameters (alpha, q, causal flag) are chosen using worst-group accuracy on a validation set, which may require group labels for model selection.
- Relies on the assumption that low-loss samples are mostly majority-group samples and on the elbow of the masking curve to separate causal parts.
Experimental setup
Datasets: Waterbirds CelebA Metashift Dominoes
Benchmarks: Waterbirds CelebA Metashift Dominoes
Metrics: worst group accuracy average group accuracy
Baselines: DFR Group DRO LISA MaskTune CnC JTT ERM
Builds on: DFR MaskTune JTT xGradCAM
Closest papers by meaning
- Let Samples Speak: Mitigating Spurious Correlation by Exploiting the Clusterness of Samples (CVPR 2025 · similarity 0.86)
- In Pursuit of Causal Label Correlations for Multi-label Image Recognition (NeurIPS 2024 · similarity 0.84)
- Discover and Mitigate Multiple Biased Subgroups in Image Classifiers (CVPR 2024 · similarity 0.84)
- Improving Group Robustness on Spurious Correlation Requires Preciser Group Inference (ICML 2024 · similarity 0.84)
- Decoupled Optimisation for Long-Tailed Visual Recognition (AAAI 2024 · similarity 0.84)
- Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuning (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.