Research map / Label-efficient and robust learning / Recognition and long-tail
ExMap: Leveraging Explainability Heatmaps for Unsupervised Group Robustness to Spurious Correlations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ExMap, which clusters LRP explainability heatmaps of a frozen ERM model to produce pseudo group labels for off-the-shelf group robustness methods (DFR, JTT) without any group annotations.
Problem
Worst-group robustness to spurious correlations normally requires group labels, which are costly to obtain; the paper targets fully unsupervised pseudo-group labeling for group robustness, including multiple-shortcut settings.
Approach
Cluster LRP explainability heatmaps (rather than penultimate-layer features as in GEORGE) with spectral clustering and an eigengap heuristic, combine clusters with class labels to form pseudo-group labels, and feed them to existing retraining methods like DFR and JTT.
Key results
- DFR+ExMap reaches 92.5% worst-group accuracy on Waterbirds versus 92.1% for supervised DFR and 76.8% for ERM, and 84.4% on CelebA versus 86.9% for DFR.
- On UrbanCars multiple-shortcut evaluation, DFR+ExMap has smaller BG+CoObj gap (-30.7) than ERM (-69.0) and GEORGE (DFR) (-63.4).
- On Waterbirds FG-Only, DFR+ExMap posts the lowest mean accuracy drop (17.5) among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| cluster explainability heatmaps instead of raw features to infer pseudo-labels | DFR · Waterbirds test | worst-group accuracy (%): 91.7 -> 92.5 | Helped |
| cluster explainability heatmaps instead of raw features to infer pseudo-labels | DFR · CelebA test | worst-group accuracy (%): 83.3 -> 84.4 | Helped |
| cluster explainability heatmaps instead of raw features to infer pseudo-labels | DFR · C-MNIST test | worst-group accuracy (%): 71.7 -> 72.5 | Helped |
| cluster explainability heatmaps instead of raw features to infer pseudo-labels | DFR · Urbancars (BG) test | worst-group accuracy (%): 69.1 -> 71.4 | Helped |
| cluster explainability heatmaps instead of raw features to infer pseudo-labels | DFR · Urbancars (CoObj) test | worst-group accuracy (%): 76.9 -> 79.2 | Helped |
| replace spectral clustering with k-means (ablation) | DFR+ExMap · Waterbirds test | worst-group accuracy (%): 92.6 -> 92.5 | No clear effect |
| replace spectral clustering with k-means (ablation) | DFR+ExMap · Waterbirds test | mean accuracy (%): 96.0 -> 95.9 | No clear effect |
| use JTT instead of DFR as the retraining strategy (ablation) | ExMap · Waterbirds test | worst-group accuracy (%): 41.1 -> 86.9 | Helped |
| apply ExMap pseudo-labels to JTT instead of using validation group labels (ablation) | JTT · Waterbirds test | worst-group accuracy (%): 86.7 -> 86.9 | No clear effect |
| use DFR with ExMap pseudo-labels rather than ERM | DFR+ExMap · Waterbirds test | worst-group accuracy (%): 76.8 -> 92.5 | Helped |
| use DFR with ExMap pseudo-labels rather than ERM | DFR+ExMap · CelebA test | worst-group accuracy (%): 41.1 -> 84.4 | Helped |
| use DFR with ExMap pseudo-labels rather than ERM | DFR+ExMap · C-MNIST test | worst-group accuracy (%): 39.6 -> 72.5 | Helped |
Surprising findings
- Heatmap-based pseudo-labels with DFR outperform the supervised DFR on Waterbirds (92.5 vs 92.1 WGA).
- k-means clustering on heatmaps performs nearly identically to spectral clustering (92.5 vs 92.6 WGA on Waterbirds).
Limitations
- Mean accuracy drops for JTT+ExMap on Waterbirds (90.0 vs 93.3), and gaps to supervised DFR remain on CelebA and other datasets.
- Relies on LRP heatmaps and a clustering step whose number of clusters is chosen by eigengap heuristic or chosen sufficiently large for k-means.
Experimental setup
Datasets: CelebA Waterbirds C-MNIST UrbanCars
Benchmarks: CelebA Waterbirds C-MNIST UrbanCars Waterbirds (FG-Only)
Metrics: worst group accuracy mean accuracy BG Gap CoObj Gap BG+CoObj Gap
Baselines: GEORGE (gDRO) GEORGE (DFR) DivDis MaskTune CVar DRO LfF JTT DFR Group DRO EIIL BARACK LLE
Builds on: DFR JTT GEORGE SPRAY Group DRO
Closest papers by meaning
- Improving Group Robustness on Spurious Correlation Requires Preciser Group Inference (ICML 2024 · similarity 0.87)
- Label-Efficient Group Robustness via Out-of-Distribution Concept Curation (CVPR 2024 · similarity 0.86)
- Rank-Constrained Adaptation for Reliable Real-World Performance (NeurIPS 2026 · similarity 0.85)
- The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlations (NeurIPS 2024 · similarity 0.85)
- Improving Subgroup Robustness via Data Selection (NeurIPS 2024 · similarity 0.85)
- Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustness (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.