Research map / Adversarial robustness and security / Attacks on text and graphs

Stage-wise Attention-Guided Region Sequencing for Adversarial Attacks on Large Vision-Language Models

NeurIPS 2026 · Adversarial robustness and security › Attacks on text and graphs · approach: Vision-language model robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SAGA, a black-box targeted attack on LVLMs that uses a single fixed cross-modal attention map from an open-source LVLM to sequence localized CLIP-surrogate perturbation updates over stage-wise attention hotspots, achieving higher ASR and imperceptibility on ten target LVLMs.

Problem

Targeted adversarial attacks on LVLMs under an L_infinity budget must decide which image regions to perturb and in what order; existing localized attacks (M-Attack, FOA-Attack) use stochastic spatial sampling over the entire image and waste budget on weakly influential regions.

Approach

Cross-modal attention from an open-source LVLM identifies adversarially sensitive hotspots, and a first-order softmax redistribution argument suggests attention released from a hotspot flows to next-salient regions; SAGA therefore schedules localized updates over precomputed stage-wise hotspots instead of random full-image crops.

Key results

• On ten target LVLMs, SAGA achieves the highest ASR across all targets, with up to 44% relative ASR gain over the strongest baseline on Gemini models (e.g., Gemini-2.5-Flash ASR 0.52 vs 0.36 for FOA-Attack).
• SAGA achieves the best imperceptibility on all six metrics (e.g., LPIPS 0.1965 vs 0.2279 for M-Attack; PSNR 29.65 vs 28.59).
• Replacing stage-wise sequencing with a single-hotspot attack lowers average ASR from 0.49 to 0.35 on four targets.

Surprising findings

• SAGA uses a smaller fraction of the perturbation budget (lower saturation ratio) yet achieves higher ASR, indicating gains come from allocation rather than stronger perturbations.
• Random and coldspot region attacks perform comparably, and ensembling attention maps from multiple extractors does not improve over the best single extractor.

Limitations

• Requires access to at least one open-source LVLM to extract the attention map.
• Not designed as a defense-robust attack; robustness to input transformations is only partially addressed with an EOT variant in the appendix.
• Attention-sensitivity correlations are modest (e.g., Pearson r=0.409 for redistribution prediction).

Experimental setup

Datasets: NIPS 2017 Adversarial Attacks and Defenses Competition dataset MSCOCO validation set

Benchmarks: NIPS 2017 adversarial competition images with MSCOCO-derived target captions

Metrics: ASR AvgSim normalized l1 normalized l2 PSNR SSIM LPIPS BRISQUE KMR

Baselines: X-Transfer AnyAttack M-Attack FOA-Attack

Builds on: M-Attack FOA-Attack transfer-based attacks with CLIP surrogates

Closest papers by meaning

• A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1 (NeurIPS 2025 · similarity 0.90)
• Pandora's Box: Towards Building Universal Attackers against Real-World Large Vision-Language Models (NeurIPS 2024 · similarity 0.89)
• Attention! Your Vision Language Model Could Be Maliciously Manipulated (NeurIPS 2025 · similarity 0.88)
• SLADE: Shielding against Dual Exploits in Large Vision-Language Models (CVPR 2025 · similarity 0.87)
• Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks (CVPR 2025 · similarity 0.87)
• Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models (CVPR 2025 · similarity 0.87)