Research map / Adversarial robustness and security / Attacks on text and graphs
Pay Less Attention to Function Words for Free Robustness of Vision-Language Models
Qiwei Tian, Chenhao Lin, Zhengyu Zhao, Chao Shen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add FDA (function-word de-attention) to fusion layers | BLIP · Flickr30k retrieval targeted attacks | avg ASR drop: +51.60% (eps 2/255), +56.36% (eps 4/255) | Improved |
| add FDA | ALBEF · RefCOCO+ grounding | avg ASR drop: 93.16/91.50 vs TeCoA 21.21/21.63 | Improved |
| add FDA vs adversarial fine-tuning (TeCoA, FARE) on clean performance | ALBEF · Flickr30k T2IR | clean R@1: 91.20 -> 95.60 | Improved |
| add FDA vs no defense on clean accuracy | ALBEF · MSCOCO T2IR | clean R@1: 77.60 -> 77.70 | No clear effect |
| add FDA vs TeCoA on stronger eps 4/255 attacks | TeCoA/FARE · Flickr30k retrieval, ALBEF | avg ASR drop: -2.15 -> 14.69 | Improved |
| add FDA to TeCoA and FARE | ALBEF · untargeted attacks retrieval | avg ASR drop: 0.47 -> 1.31 (TeCoA), 13.09 -> 13.46 (FARE) | Improved |
| apply FDA in fusion encoder only rather than text encoder only (ablation) | FDA · Flickr30k retrieval, ALBEF | avg ASR drop: -2.54 -> 18.48 | Improved |
| apply FDA in both text and fusion encoders instead of fusion only (ablation) | FDA · Flickr30k retrieval, ALBEF | avg ASR drop / clean: 18.48 -> 15.61 | Hurt |
| use shortlisted 91-word dictionary instead of full 208-word dictionary (ablation) | FDA · Flickr30k retrieval, ALBEF | avg ASR drop: 4.22 -> 6.45 | Improved |
| apply FDA in shallow layers (L0-1) instead of all layers (ablation) | FDA · Flickr30k retrieval, BLIP | avg ASR drop: 26.46 -> 53.98 | Improved |
| use FDA instead of directly masking function words (ablation) | ALBEF · Flickr30k retrieval | avg ASR drop: 1.56 -> 23.07 | Improved |
| de-attend determiners or adjectives instead of all function words (ablation) | FDA · Flickr30k retrieval, ALBEF | avg ASR drop: 23.07 -> 9.28 (DDA), 15.10 (ADA) | Hurt |
Closest papers by meaning
- One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models (CVPR 2024 · similarity 0.87)
- Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models (ICML 2024 · similarity 0.86)
- ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.86)
- R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning (CVPR 2025 · similarity 0.86)
- Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models (NeurIPS 2024 · similarity 0.85)
- Jailbreaking Vision-Language Models via Dissonance-Guided Suffix Optimization and Image-Phrase Injection (CVPR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
