Research map / Adversarial robustness and security / Black-box and transfer attacks
Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FOA-Attack, a targeted transferable attack on MLLMs that combines a cosine global [CLS] loss, a clustered local optimal-transport loss over patch tokens, and dynamic ensemble weighting across CLIP encoders.
Problem
Targeted transfer-based adversarial attacks against MLLMs, especially closed-source ones, are limited because existing methods align only global [CLS] features and ignore local patch-token information.
Approach
Align adversarial and target images at both global ([CLS]) and local levels: K-means cluster patch tokens into compact centers and align them via a Sinkhorn optimal transport loss, plus a loss-decrease-rate-based dynamic weighting of the CLIP ensemble.
Key results
- On closed-source GPT-4o and GPT-4.1, FOA-Attack reaches 75.1% and 77.3% ASR at threshold 0.5, outperforming M-Attack by 14.8 and 16.5 points.
- On Gemini-2.0, FOA-Attack achieves 53.4% ASR and 0.50 AvgSim versus below 8% for other baselines other than M-Attack's 44.8%.
- On open-source LLaVA-1.5-7B, FOA-Attack averages 79.6% ASR versus 68.3% for M-Attack.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FOA-Attack instead of M-Attack | targeted transfer attack on Qwen2.5-VL-7B · Qwen2.5-VL-7B | ASR: 52.6 -> 70.7 | Helped |
| use FOA-Attack instead of M-Attack | targeted transfer attack on LLaVa-1.5-7B · LLaVa-1.5-7B | ASR: 68.3 -> 79.6 | Helped |
| use FOA-Attack instead of M-Attack | targeted transfer attack on GPT-4o · GPT-4o | ASR: 60.3 -> 75.1 | Helped |
| use FOA-Attack instead of M-Attack | targeted transfer attack on GPT-4.1 · GPT-4.1 | ASR: 60.8 -> 77.3 | Helped |
| use FOA-Attack instead of M-Attack | targeted transfer attack on GPT-o3 · GPT-o3 | ASR: 67.0 -> 81.0 | Helped |
| add global feature alignment (ablation) | FOA-Attack · GPT-4o | ASR: 78.0 -> 81.0 | Helped |
| add global feature alignment (ablation) | FOA-Attack · Claude-3.5 | ASR: 14.0 -> 16.0 | Helped |
| add local feature alignment (ablation) | FOA-Attack · GPT-4o | ASR: 76.0 -> 81.0 | Helped |
| add local feature alignment (ablation) | FOA-Attack · Claude-3.5 | ASR: 12.0 -> 16.0 | Helped |
| add dynamic loss weighting (ablation) | FOA-Attack · GPT-4o | ASR: 79.0 -> 81.0 | Helped |
| add dynamic loss weighting (ablation) | FOA-Attack · Claude-3.5 | ASR: 13.0 -> 16.0 | Helped |
| increase cluster centers from [3] to [3,5] (ablation) | FOA-Attack · Claude-3.5 | ASR: 12.0 -> 16.0 | Helped |
Surprising findings
- Reasoning-enhanced closed-source MLLMs appear no more robust, and sometimes weaker, than non-reasoning ones to these attacks.
Limitations
- Progressive multi-cluster strategy substantially increases attack time.
- Closed-source transfer remains low on some models such as Claude-3.5/3.7 (about 12-16% ASR).
Experimental setup
Datasets: NIPS 2017 Adversarial Attacks and Defenses competition images MSCOCO validation set
Benchmarks: NIPS 2017 adversarial competition images vs MSCOCO targets
Metrics: attack success rate AvgSim KMR
Baselines: AttackVLM AdvDiffVLM SSA-CWA AnyAttack M-Attack
Builds on: M-Attack AttackVLM Sinkhorn algorithm Common Weakness Attack
Closest papers by meaning
- Towards Transferable Adversarial Attacks with Centralized Perturbation (AAAI 2024 · similarity 0.88)
- Boosting Adversarial Transferability through Augmentation in Hypothesis Space (CVPR 2025 · similarity 0.88)
- Rethinking Model Ensemble in Transfer-based Adversarial Attacks (ICLR 2024 · similarity 0.87)
- Boosting Adversarial Transferability with Spatial Adversarial Alignment (NeurIPS 2025 · similarity 0.87)
- Enhancing Adversarial Transferability with Adversarial Weight Tuning (AAAI 2025 · similarity 0.87)
- Understanding Model Ensemble in Transferable Adversarial Attack (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.