Research map / Label-efficient and robust learning / Recognition and long-tail
Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Project-Probe-Aggregate (PPA), a parameter-efficient linear-probe method that projects CLIP features onto the null space of text class proxies to get a biased model, infers pseudo-groups, trains with group logit adjustment, and aggregates group weights into a debiased classifier without group labels.
Problem
Improving worst-group accuracy of image-text foundation models under spurious correlations without group annotations and without costly full fine-tuning.
Approach
Remove class-proxy directions (from CLIP text embeddings) from image features to induce a stronger spurious-feature biased model for pseudo-group identification, then train a group-level linear probe with logit-adjusted softmax (minimizing balanced group error) and sum group weights per class into a single debiased classifier.
Key results
- With CLIP ResNet-50, PPA reaches 84.3% worst-group accuracy on Waterbirds, 91.1% on CelebA and 90.8% on MetaShift, improving over CFR by 7.4, 17.4 and 9.3 points.
- On Waterbirds training set, PPA achieves 80.4% worst-group recall and 44.3% precision for identifying minority groups, versus 78.6/24.1 for JTT and 46.4/15.6 for CA.
- PPA uses under 0.1% (text claims <0.01% in abstract) trainable parameters (4,096 for the ResNet-50 linear head).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Project image features to remove class proxies (ablation) | PPA group identification · Waterbirds | worst-group accuracy: 81.6 -> 84.3 | Helped |
| Apply group logit adjustment for prior correction (ablation) | PPA group-target training with projection · Waterbirds | worst-group accuracy: 81.6 -> 84.3 | Helped |
| Replace inferred group labels with ground-truth group labels (ablation) | PPA · Waterbirds | worst-group accuracy: 84.3 -> 86.8 | Helped |
| Replace inferred group labels with ground-truth group labels (ablation) | PPA · CelebA | worst-group accuracy: 91.1 -> 91.5 | Helped |
| Replace inferred group labels with ground-truth group labels (ablation) | PPA · MetaShift | worst-group accuracy: 90.8 -> 91.3 | Helped |
| Use PPA instead of CA | CLIP ResNet-50 · Waterbirds | worst-group accuracy: 83.7 -> 84.3 | Helped |
| Use PPA instead of CA | CLIP ResNet-50 · CelebA | worst-group accuracy: 90.0 -> 91.1 | Helped |
| Use PPA instead of Zero-Shot | CLIP ResNet-50 · MetaShift | worst-group accuracy: 86.2 -> 90.8 | Helped |
| Use PPA instead of CFR | CLIP ViT-L/14 · Waterbirds | worst-group accuracy: 88.2 -> 87.2 | Hurt |
| Use PPA instead of CA | CLIP ViT-L/14 · CelebA | worst-group accuracy: 84.6 -> 90.4 | Helped |
| Use PPA instead of Group Prompt ZS | CLIP ViT-L/14 · MetaShift | worst-group accuracy: 93.9 -> 94.8 | Helped |
| Use PPA instead of CA | CLIP ResNet-50 · Living-17 | worst-group accuracy: 62.0 -> 65.0 | Helped |
Surprising findings
- Removing class information from image features (projection) amplifies reliance on spurious features, which improves minority group identification.
Limitations
- Modest drop in average (in-distribution) accuracy, about 1.8 points on average versus ERM with CLIP ViT-L/14.
- Relies on pseudo-group quality; gap to ground-truth group labels remains (e.g., 84.3 vs 86.8 on Waterbirds).
Experimental setup
Datasets: Waterbirds CelebA MetaShift BAR Living-17
Benchmarks: Waterbirds CelebA MetaShift Living-17 BAR
Metrics: worst-group accuracy average accuracy worst-group precision worst-group recall
Baselines: Zero-shot CLIP Group prompt ZS ERM WiSE-FT Orth-Cali AFR JTT CnC CA CFR GroupDRO DFR
Builds on: JTT Learning from Failure (LfF) logit adjustment loss CLIP Last-layer retraining (DFR)
Closest papers by meaning
- Label-Efficient Group Robustness via Out-of-Distribution Concept Curation (CVPR 2024 · similarity 0.87)
- The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlations (NeurIPS 2024 · similarity 0.86)
- Discover and Mitigate Multiple Biased Subgroups in Image Classifiers (CVPR 2024 · similarity 0.85)
- Group-robust Sample Reweighting for Subpopulation Shifts via Influence Functions (ICLR 2025 · similarity 0.84)
- Improving Group Robustness on Spurious Correlation Requires Preciser Group Inference (ICML 2024 · similarity 0.84)
- ExMap: Leveraging Explainability Heatmaps for Unsupervised Group Robustness to Spurious Correlations (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.