Research map / Adversarial robustness and security / Attacks on text and graphs
Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization
Xiang Fang, Wanlong Fang, Changshuo Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add text attack (ablation) | MMAS · DALLE-3, LLaVA | Overall semantic similarity: 0.693 -> 0.872 | Improved |
| add image attack (ablation) | MMAS · DALLE-3, LLaVA | Overall semantic similarity: 0.681 -> 0.872 | Improved |
| add joint optimization of image and text perturbations (ablation) | MMAS · DALLE-3, LLaVA | Overall semantic similarity: 0.793 -> 0.872 | Improved |
| add texture constraint and cross-modal optimization over TC-UAP | TC-UAP · MS-COCO, LLaVA | Overall semantic similarity: 0.803 -> 0.883 | Improved |
| use MMAS instead of TA-UAP | TA-UAP · MS-COCO, BLIP-2 | Overall semantic similarity: 0.799 -> 0.849 | Improved |
| raise texture scaling factor from 1 to 4 (ablation) | MMAS · DALLE-3, LLaVA-1.5 | Overall semantic similarity: 0.823 -> 0.872 | Improved |
| raise texture scaling factor from 4 to 8 (ablation) | MMAS · DALLE-3, LLaVA-1.5 | Overall semantic similarity: 0.872 -> 0.841 | Hurt |
| use MMAS instead of MF-Attack | MF-Attack · ImageNet-1k VQA | Mean semantic similarity: 0.646 -> 0.839 | Improved |
| use MMAS instead of CroPA | CroPA · MS-COCO OpenFlamingo | Overall ASR: 0.83 -> 0.89 | Improved |
| transfer MMAS adversarial examples instead of CroPA to GPT-4o | CroPA · LLaVA to GPT-4o | ASR: 0.011 -> 0.379 | Improved |
| use MMAS instead of CroPA for generation cost | CroPA · LLaVA-1.5 | GPU time: 17 min -> 6 min | Improved |
| change target text among six alternatives (ablation) | MMAS · MS-COCO, Flamingo | Overall semantic similarity: 0.868 to 0.895 | No clear effect |
Closest papers by meaning
- Pandora's Box: Towards Building Universal Attackers against Real-World Large Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- SLADE: Shielding against Dual Exploits in Large Vision-Language Models (CVPR 2025 · similarity 0.90)
- On the Robustness of Large Multimodal Models Against Image Adversarial Attacks (CVPR 2024 · similarity 0.90)
- Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (ICLR 2024 · similarity 0.89)
- Mutual-Modality Adversarial Attack with Semantic Perturbation (AAAI 2024 · similarity 0.89)
- ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
