Research map / Adversarial robustness and security / Attacks on text and graphs
Text-Guided Gradient Refinement: Resolving Multimodal Gradient Conflicts to Boost Adversarial Attacks on Vision-Language Models
Yuyang Huang, Tianzuo Luo, Hengyuan Guo, Yuren Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fuse image-target and text-target gradients linearly instead of image-only (ablation) | M-Attack (image-only guidance) · GPT-4o | ASR (%): 82% -> 92% | Improved |
| fuse image-target and text-target gradients linearly instead of text-only (ablation) | text-only guidance · GPT-4o | ASR (%): 74% -> 92% | Improved |
| add conflict-aware gradient projection (remove image gradient component opposing text gradient) (ablation) | naive gradient fusion · GPT-4o | ASR (%): 92% -> 96% | Improved |
| add conflict-aware gradient projection (ablation) | naive gradient fusion · GPT-4o | CLIP score (ensemble): 58.95 -> 61.01 | Improved |
| use image-only guidance instead of text-only guidance (ablation) | gradient source · GPT-4o | ASR (%): 74% -> 82% | Improved |
| use balanced fusion coefficient alpha 0.7 rather than text-only (0) or image-only (1.0) (ablation) | gradient fusion · BLIP-2 (multi-surrogate) | CLIP score | Improved |
| use multi-surrogate ensemble instead of single CLIP surrogate | AttackVLM-style attacks · GPT-4o | ASR (%) | Improved |
| add TGGR to M-Attack | M-Attack · LLaVA-13B | ASR (%): 84% -> 90% | Improved |
| add TGGR to M-Attack | M-Attack · BLIP-2 | ASR (%): 80% -> 82% | Improved |
| image-gradient and text-gradient conflict grows during optimization | fused gradient (alpha 0.7) · gradient analysis | conflict percentage: 15.5% -> 83.5% | Hurt |
Closest papers by meaning
- SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environments (CVPR 2025 · similarity 0.90)
- Failures to Find Transferable Image Jailbreaks Between Vision-Language Models (ICLR 2025 · similarity 0.90)
- A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1 (NeurIPS 2025 · similarity 0.89)
- Ask, Attend, Attack: An Effective Decision-Based Black-Box Targeted Attack for Image-to-Text Models (NeurIPS 2024 · similarity 0.89)
- Jailbreaking Vision-Language Models via Dissonance-Guided Suffix Optimization and Image-Phrase Injection (CVPR 2026 · similarity 0.89)
- SLADE: Shielding against Dual Exploits in Large Vision-Language Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
