Research map / Adversarial robustness and security / Attacks on text and graphs
Multi-Faceted Attack: Exposing Cross-Model Vulnerabilities in Defense-Equipped Vision-Language Models
Yijun Yang, Lichao Wang, Jianping Zhang, Chi Harold Liu, Lanqing Hong, Qiang Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Attention-Transfer Attack (meta task asking two contrasting answers) alone (ablation) | MFA on open-source VLMs · HEHS (6 open-source VLMs) | ASR: 17.5 -> 63.33 (avg ASR %) | Improved |
| add vision-encoder adversarial image alone (ablation) | MFA on open-source VLMs · HEHS (6 open-source VLMs) | ASR: 17.5 -> 59.58 (avg ASR %) | Improved |
| add filter-targeted adversarial signature alone (ablation) | MFA on open-source VLMs · HEHS (6 open-source VLMs) | ASR: 17.5 -> 20.00 (avg ASR %) | No clear effect |
| combine all three facets (ablation) | MFA · HEHS (6 open-source VLMs) | ASR: 63.33 -> 72.92 (best single facet vs MFA, avg ASR %) | Improved |
| add weakly supervised transfer optimization to multi-token signature search (ablation) | filter-targeted attack · HEHS (7 moderators) | ASR: 67.50 -> 80.00 (avg %) | Improved |
| add weakly supervised transfer optimization to multi-token signature search (ablation) | filter-targeted attack · StrongReject (7 moderators) | ASR: 60.18 -> 68.70 (avg %) | Improved |
| use multi-token optimization instead of GCG (ablation) | filter-targeted attack · HEHS (7 moderators) | ASR: 59.11 -> 67.50 (avg %) | Improved |
| use MFA instead of GPTFuzzer | MFA · HEHS, GPT-4.1 | LlamaGuard harmful rate: 0.0 -> 40.0 | Improved |
| use MFA instead of GPTFuzzer | MFA · HEHS, LLaMA-3.2 and LLaMA-4-Scout | harmful rate | Hurt |
| use rephrased meta-task prompt variants (ablation) | ATA · HEHS, Grok-2-Vision | harmful rate: 90.0 -> 80.0-90.0 | No clear effect |
| use MFA instead of best baselines | MFA · HEHS, avg over 17 VLMs | human ASR: 54.3 -> 58.5 | Improved |
Closest papers by meaning
- Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models (CVPR 2025 · similarity 0.91)
- ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.91)
- Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks (CVPR 2025 · similarity 0.90)
- JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models (NeurIPS 2025 · similarity 0.90)
- A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1 (NeurIPS 2025 · similarity 0.90)
- Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
