Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Zero-Shot Robustification of Zero-Shot Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RoboShot, a fully zero-shot method that uses LM-generated harmful and helpful insights to project pretrained embeddings (removing spurious and amplifying useful directions), plus a label-free adapter variant.
Problem
Zero-shot models inherit spurious correlations that hurt worst-group accuracy, and existing robustness fixes need labeled data and fine-tuning or manual concept specification.
Approach
Obtain harmful and helpful concept directions from language model answers to task-description prompts, embed them, and in embedding space reject harmful components and amplify helpful components of image/text embeddings, with no labels or training; a label-free adaptation learns a projection initialized from this.
Key results
- Across nine image and NLP datasets, RoboShot reports an average 15.98% worst-group accuracy improvement with trivial average accuracy change over zero-shot baselines.
- On Waterbirds with CLIP ViT-B-32, worst-group accuracy rises from 27.9 (ZS) to 54.4 with AVG 82.0 vs 80.7.
- On CivilComments with BERT, worst-group accuracy improves from 33.3 to 42.3 and gap drops from 14.8 to 7.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RoboShot's harmful-component rejection and helpful-component boosting | CLIP ViT-B-32 zero-shot classification · Waterbirds | worst-group accuracy: 27.9 -> 54.4 | Helped |
| Apply RoboShot's harmful-component rejection and helpful-component boosting | CLIP ViT-B-32 zero-shot classification · CelebA | worst-group accuracy: 72.7 -> 80.5 | Helped |
| Apply RoboShot's harmful-component rejection and helpful-component boosting | CLIP ViT-B-32 zero-shot classification · PACS | worst-group accuracy: 82.1 -> 86.3 | Helped |
| Apply RoboShot's harmful-component rejection and helpful-component boosting | CLIP ViT-B-32 zero-shot classification · VLCS | worst-group accuracy: 20.5 -> 33.0 | Helped |
| Apply RoboShot's harmful-component rejection and helpful-component boosting | BiomedCLIP zero-shot classification · CXR14 | worst-group accuracy: 28.9 -> 41.6 | Helped |
| Reject harmful insight vectors | BERT zero-shot classification · HateXplain | worst-group accuracy: 0.0 -> 14.0 | Helped |
| Reject harmful insight vectors | BERT zero-shot classification · Amazon | worst-group accuracy: 64.2 -> 64.4 | Helped |
| Use harmful-vector rejection only rather than the full method (ablation) | RoboShot with CLIP ViT-B-32 · Waterbirds | worst-group accuracy: 54.4 -> 50.4 | Hurt |
| Use helpful-vector boosting only rather than the full method (ablation) | RoboShot with CLIP ViT-B-32 · Waterbirds | worst-group accuracy: 54.4 -> 30.2 | Hurt |
| Use harmful-vector rejection only rather than the full method (ablation) | RoboShot with CLIP ViT-B-32 · CelebA | worst-group accuracy: 80.5 -> 81.5 | Hurt |
| Use helpful-vector boosting only rather than the full method (ablation) | RoboShot with CLIP ViT-B-32 · CelebA | worst-group accuracy: 80.5 -> 71.3 | Hurt |
| Use harmful-vector rejection only rather than the full method (ablation) | RoboShot with CLIP ViT-B-32 · PACS | worst-group accuracy: 86.3 -> 83.7 | Hurt |
Surprising findings
- Weaker LMs (GPT2, Flan-T5) still give insights that improve over zero-shot baselines on several datasets, suggesting precision of insights matters more than LM strength.
Limitations
- In text experiments only harmful-component rejection is used because harmful and helpful insights are less distinct.
- Rare cases where the method does not improve baseline (e.g., ALIGN on Waterbirds), and LFA requires a small labeled validation set.
- Theory relies on a simplified Gaussian noise model with orthonormal concept assumptions.
Experimental setup
Datasets: Waterbirds CelebA CXR14 PACS VLCS CivilComments-WILDS HateXplain Amazon-WILDS Gender Bias
Benchmarks: Waterbirds CelebA CXR14 PACS VLCS CivilComments-WILDS HateXplain Amazon-WILDS Gender Bias
Metrics: average accuracy worst-group accuracy gap between AVG and WG
Baselines: ZS Group Prompt ZS Direct prompting BART-MNLI ChatGPT CLIP ALIGN AltCLIP
Builds on: Group Prompt ZS (spurious descriptors) embedding debiasing methods WILDS
Closest papers by meaning
- Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions (CVPR 2024 · similarity 0.89)
- OTTER: Effortless Label Distribution Adaptation of Zero-shot Models (NeurIPS 2024 · similarity 0.89)
- LR0.FM: Low-Res Benchmark and Improving robustness for Zero-Shot Classification in Foundation Models (ICLR 2025 · similarity 0.89)
- TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning (NeurIPS 2024 · similarity 0.88)
- On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach (CVPR 2025 · similarity 0.87)
- Descriptor and Word Soups: Overcoming the Parameter Efficiency Accuracy Tradeoff for Out-of-Distribution Few-shot Learning (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.