Research map / Vision-language and multimodal / CLIP and zero-shot
SAGE: Spuriousness-Aware Guided Prompt Exploration for Mitigating Multimodal Bias
Wenqian Ye, Di Wang, Guangtao Zheng, Bohan Liu, Aidong Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select a prompt per image by separation score (top-1) instead of fixed default prompt | zero-shot CLIP models · Waterbirds (5-backbone avg) | worst-group accuracy: 36.7 -> 44.9 | Improved |
| select a prompt per image by separation score on CelebA | zero-shot CLIP models · CelebA (5-backbone avg) | worst-group accuracy: 75.3 -> 80.6 | Improved |
| SAGE on PACS and VLCS | zero-shot CLIP models · PACS / VLCS (5-backbone avg) | worst-group accuracy: 75.5 -> 81.9 (PACS); 23.0 -> 33.8 (VLCS) | Improved |
| SAGE average accuracy on VLCS vs zero-shot | zero-shot CLIP models · VLCS (5-backbone avg) | average accuracy: 76.1 -> 75.8 | No clear effect |
| use separation-score selection instead of random prompt selection (ablation) | SAGE · Waterbirds (avg) | worst-group accuracy: 40.1 -> 44.9 | Improved |
| use separation-score selection instead of random prompt selection (ablation) | SAGE · CelebA (avg) | worst-group accuracy: 74.9 -> 80.6 | Improved |
| use top-1 selected prompt instead of ensembling all 80 prompts (ablation) | SAGE · Waterbirds (avg) | worst-group accuracy: 36.2 -> 44.9 | Improved |
| use top-1 selected prompt instead of ensembling all 80 prompts (ablation) | SAGE · CelebA (avg) | worst-group accuracy: 73.2 -> 80.6 | Improved |
| increase number of selected prompts K above 1 (ablation) | SAGE · Waterbirds, CelebA | worst-group accuracy | Mixed |
| SAGE on CLIP-RN-50 Waterbirds | CLIP-RN-50 · Waterbirds | worst-group accuracy: 41.0 -> 41.3 | No clear effect |
Closest papers by meaning
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.89)
- Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations (CVPR 2024 · similarity 0.89)
- Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting (NeurIPS 2024 · similarity 0.88)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.88)
- CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation (CVPR 2025 · similarity 0.88)
- A Sober Look at the Robustness of CLIPs to Spurious Features (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
