Research map / Vision-language and multimodal / CLIP and zero-shot
FALIP: Visual Prompt as Foveal Attention Boosts CLIP Zero-Shot Performance
Jiedong Zhuang, Jiaqi Hu, Lianrui Mu, Rui Hu, Xiaoyu Liang, Jiangnan Ye, Haoji Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| insert foveal attention mask on the [CLS] row (Method a) instead of all rows (ablation) | FALIP · REC average (RefCOCO family) | average accuracy: 39.3 -> 45.2 | Improved |
| insert foveal attention mask on the [CLS] row instead of the diagonal (ablation) | FALIP · REC average (RefCOCO family) | average accuracy: 15.8 -> 45.2 | Improved |
| use attention mask instead of feature mask (ablation) | FALIP · REC average (RefCOCO family) | average accuracy: 27.0 -> 45.2 | Improved |
| add foveal attention mask to original CLIP (ablation) | CLIP ViT-B/16 · REC average (RefCOCO family) | average accuracy: 21.5 -> 45.2 | Improved |
| apply RedCircle visual prompt to CLIP for fine-grained classification | CLIP ViT-B/16 · CUB-200-2011 | Top1: 54.2 -> 44.2 | Hurt |
| apply Blur visual prompt to CLIP | CLIP ViT-B/16 · Waterbirds | Top1: 78.2 -> 78.1 | No clear effect |
| use FALIP instead of original CLIP for classification | CLIP ViT-B/16 · CUB-200-2011 | Top1: 54.2 -> 54.3 | No clear effect |
| use FALIP instead of Blur on Waterbirds with ViT-L | CLIP ViT-L/14 · Waterbirds (ViT-L) | Top1: 85.1 -> 84.5 | Hurt |
| unleash salient attention heads in last 4 layers for RedCircle (ablation) | RedCircle · REC average (RefCOCO family) | average accuracy: 41.3 -> 45.2 | Improved |
| insert mask in layers 9 to 12 instead of layers 1 to 4 (ablation) | FALIP · REC average (RefCOCO family) | average accuracy: 24.0 -> 45.2 | Improved |
| use alpha=0.2 and sigma=100 instead of sigma=1 (ablation) | FALIP · REC average (RefCOCO family) | average accuracy: 39.5 -> 45.2 | Improved |
| add FALIP on top of RedCircle with ensemble and post-processing | RedCircle · RefCOCO TestA (gold, with E and P) | accuracy: 53.5 -> 53.4 | Mixed |
Closest papers by meaning
- Tune-An-Ellipse: CLIP Has Potential to Find What You Want (CVPR 2024 · similarity 0.88)
- Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting (NeurIPS 2024 · similarity 0.87)
- PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts (ICLR 2024 · similarity 0.87)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.86)
- DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception (CVPR 2025 · similarity 0.85)
- VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
