Research map / Adversarial robustness and security / Black-box and transfer attacks
Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Chain of Attack (CoA), a transfer-based targeted adversarial image attack on VLMs using modality-aware CLIP embeddings, iterative caption-based semantic updates, and a contrastive matching loss, plus an LLM-judged attack success rate.
Problem
Black-box transfer-based adversarial attacks on open-source vision-language models generate targeted responses, but existing methods ignore vision-text semantic correspondence and lack a unified, human-interpretable attack success rate for free-form response generation.
Approach
Crafts adversarial images via a chain of intermediate steps: at each PGD step, an image-to-text model captions the current adversarial image and the modality-aware embedding (fused CLIP image and text embeddings) is updated, with a Targeted Contrastive Matching loss pulling it toward the target reference and away from the clean example; success is scored by an LLM judge (1/0.5/0).
Key results
- On UniDiffuser, CoA reaches 94.2% targeted ASR and 98.9% fooled ASR versus 90.0% and 98.4% for the second-best MF-ii.
- On ViECap, CoA reaches 98.4% targeted ASR versus 76.6% for MF-ii.
- On LLaVA-7B, CoA reaches 14.5% targeted ASR and 28.4% fooled ASR, showing larger VLMs are harder to attack.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoA instead of MF-it | ViECap · ViECap | Ensemble CLIP score: 77.0 -> 82.3 | Helped |
| Use CoA instead of MF-ii | ViECap · ViECap | Target ASR: 76.6 -> 98.4 | Helped |
| Use CoA instead of MF-ii | SmallCap · SmallCap | Ensemble CLIP score: 65.8 -> 67.2 | Helped |
| Use CoA instead of MF-ii | SmallCap · SmallCap | Target ASR: 47.1 -> 56.8 | Helped |
| Use CoA instead of MF-ii | Unidiffuser · Unidiffuser | Ensemble CLIP score: 70.0 -> 75.2 | Helped |
| Use CoA instead of MF-ii | Unidiffuser · Unidiffuser | Target ASR: 90.0 -> 94.2 | Helped |
| Use CoA instead of MF-ii | LLaVA-7B · LLaVA-7B | Ensemble CLIP score: 44.9 -> 48.7 | Helped |
| Use CoA instead of MF-ii | LLaVA-7B · LLaVA-7B | Target ASR: 2.6 -> 14.5 | Helped |
| Use CoA instead of MF-ii | LLaVA-13B · LLaVA-13B | Ensemble CLIP score: 45.2 -> 45.8 | Helped |
| Use CoA instead of MF-ii | LLaVA-13B · LLaVA-13B | Target ASR: 3.6 -> 12.3 | Helped |
| Add modality-aware embeddings (ablation) | Unidiffuser baseline · Unidiffuser | Ensemble CLIP score: 70.0 -> 71.3 | Helped |
| Add Chain of Attack without TCM after modality-aware embeddings (ablation) | Unidiffuser · Unidiffuser | Ensemble CLIP score: 71.3 -> 73.8 | Helped |
Surprising findings
- Larger VLMs (LLaVA-13B) are substantially harder to target, with CLIP scores near the clean-image baseline for several attacks.
Limitations
- Attack is evaluated only on image captioning-style responses for a limited set of victim models.
- The LLM judge (GPT-4) is a closed model whose scoring reliability is not independently validated.
Experimental setup
Datasets: ImageNet-1K validation MS-COCO captions
Benchmarks: ImageNet-1K validation (clean images) with ViECAP, SmallCap, UniDiffuser, LLaVA-1.5 7B/13B victims
Metrics: CLIP score LLM-based attack success rate (targeted and fooled) LPIPS
Baselines: VIECAP AttackBard Mix.Attack MF-it MF-ii MF-ii-tt
Builds on: MF-ii Mix.Attack AttackBard CLIP PGD
Closest papers by meaning
- Attention! Your Vision Language Model Could Be Maliciously Manipulated (NeurIPS 2025 · similarity 0.91)
- A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1 (NeurIPS 2025 · similarity 0.90)
- Retention Score: Quantifying Jailbreak Risks for Vision Language Models (AAAI 2025 · similarity 0.89)
- Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models (CVPR 2025 · similarity 0.89)
- Pandora's Box: Towards Building Universal Attackers against Real-World Large Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.