Research map / Vision-language and multimodal / CLIP and zero-shot
ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ERICT, a training-free inference-time method that uses auxiliary text prompts to mask spurious image tokens in the CLIP vision encoder's CLS attention, improving worst-group accuracy on spurious-correlation benchmarks.
Problem
Zero-shot vision-language models such as CLIP rely on spurious correlations (e.g., backgrounds, gender), hurting worst-group robustness; existing fixes need group labels, fine-tuning, or LLM-generated prompts.
Approach
Use an auxiliary text prompt (a superclass template or aggregated class embeddings) to score image patch tokens, sparsemax-style binarize them into a token mask, and apply the mask to the CLS token's attention in the last layers so the CLS representation ignores spurious patches, with no training, LLMs, or group labels.
Key results
- On CelebA with ViT-L/14, ERICT-C reaches 85.44% worst-group accuracy and a 1.09 gap, versus 72.77% for zero-shot CLIP.
- On Urbancars with ViT-L/14, ERICT raises worst-group accuracy from 12.4% to 49.6%.
- On Waterbirds with ViT-B/32, ERICT achieves 71.44% worst-group accuracy versus 39.33% for zero-shot CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ERICT instead of PerceptionCLIP | ViT-B/32 · Waterbirds | worst-group accuracy: 66.07 -> 71.44 | Helped |
| Use ERICT instead of PerceptionCLIP+ | ViT-B/16 · Waterbirds | worst-group accuracy: 41.07 -> 56.23 | Helped |
| Use ERICT-C instead of PerceptionCLIP+ | ViT-L/14 · Waterbirds | worst-group accuracy: 61.12 -> 62.31 | Helped |
| Use ERICT-C instead of ROBOSHOT | ViT-B/32 · CelebA | worst-group accuracy: 80.5 -> 83.62 | Helped |
| Use ERICT-C instead of PerceptionCLIP+ | ViT-B/16 · CelebA | worst-group accuracy: 69.18 -> 81.98 | Helped |
| Use ERICT-C instead of ROBOSHOT | ViT-L/14 · CelebA | worst-group accuracy: 82.6 -> 85.44 | Helped |
| Use ERICT instead of Group Prompt | ViT-B/32 · Urbancars | worst-group accuracy: 33.20 -> 38.80 | Helped |
| Use ERICT instead of Group Prompt | ViT-B/16 · Urbancars | worst-group accuracy: 7.60 -> 9.20 | Helped |
| Use ERICT instead of Group Prompt | ViT-L/14 · Urbancars | worst-group accuracy: 25.20 -> 49.60 | Helped |
| Apply the mask to the image rather than use unmodified ZSCLIP (ablation) | ViT-L/14 · CelebA | worst-group accuracy: 72.77 -> 81.67 | Helped |
| Apply the mask to the last 1 layer rather than use unmodified ZSCLIP (ablation) | ViT-L/14 · CelebA | worst-group accuracy: 72.77 -> 83.33 | Helped |
| Apply the mask to the last 3 layers instead of the last 2 layers (ablation) | ERICT, ViT-L/14 · CelebA | worst-group accuracy: 85.44 -> 84.44 | Hurt |
Surprising findings
- ERICT does not show significant improvement in some cases, and ERICT-C outperforms ERICT in many cases, indicating the auxiliary prompt is suboptimal.
Limitations
- Auxiliary prompt choice (superclass template or class aggregation) is suboptimal and not guaranteed optimal.
- Performance gains are inconsistent across settings; ERICT-C often beats ERICT.
- Temperature must be tuned per dataset, with the optimal value varying by dataset.
Experimental setup
Datasets: Waterbirds CelebA Urbancars ImageNet (appendix)
Benchmarks: Waterbirds CelebA Urbancars
Metrics: worst-group accuracy average accuracy gap between average and worst-group accuracy
Baselines: ZSCLIP Group Prompt PerceptionCLIP PerceptionCLIP+ ROBOSHOT
Builds on: CLIP sparsemax PerceptionCLIP ROBOSHOT
Closest papers by meaning
- The Neglected Tails in Vision-Language Models (CVPR 2024 · similarity 0.88)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.88)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.87)
- Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning (ICML 2024 · similarity 0.87)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.