Research map / Adversarial robustness and security / Black-box and transfer attacks
Ask, Attend, Attack: An Effective Decision-Based Black-Box Targeted Attack for Image-to-Text Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AAA, a decision-based black-box targeted attack on image-to-text models that uses evolutionary search over pixels, guided by a target-semantic dictionary (Ask) and a Grad-CAM attention mask from a surrogate ResNet (Attend), to attack without semantic loss.
Problem
Targeted adversarial attacks on image-to-text models when the attacker can only observe the model's output text (decision-based black-box), where prior gray-box methods suffer semantic loss.
Approach
Formulates the black-box targeted attack as a large-scale optimization over pixels solved by differential evolution, with a target text chosen from a semantic dictionary built during search (Ask), the search space restricted by a surrogate-model Grad-CAM heatmap (Attend), and CLIP text-embedding distance as fitness (Attack).
Key results
- On VIT-GPT2 at epsilon 25, AAA reaches METEOR 0.696, BLEU 0.658, CLIP 0.952, SPICE 0.634, versus 0.433/0.411/0.832/0.35 for the best gray-box baseline transfer+query.
- On Show-Attend-Tell at epsilon 25, AAA reaches CLIP 0.964 and SPICE 0.786, outperforming the gray-box transfer+query baseline (CLIP 0.794, SPICE 0.311).
- Removing Attend drops AAA at epsilon 25 on VIT-GPT2 to METEOR 0.541, and AAA takes 151 seconds per sample versus 29 to 97 seconds for gray-box baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Attend attention heatmap to reduce the search space (ablation) | AAA · Flick30k, ε=25; VIT-GPT2 | METEOR: 0.541±0.25 -> 0.696±0.21 | Helped |
| Add Ask semantic-dictionary target text instead of random words (ablation) | AAA · Flick30k, ε=25; VIT-GPT2 | METEOR: 0.398±0.21 -> 0.696±0.21 | Helped |
| Add Attend attention heatmap to reduce the search space (ablation) | AAA · Flick30k, ε=25; Show-Attend-Tell | METEOR: 0.642±0.19 -> 0.855±0.15 | Helped |
| Add Ask semantic-dictionary target text instead of random words (ablation) | AAA · Flick30k, ε=25; Show-Attend-Tell | METEOR: 0.364±0.21 -> 0.855±0.15 | Helped |
| Add Attend attention heatmap to reduce the search space (ablation) | AAA · Flick30k, ε=15; VIT-GPT2 | METEOR: 0.461±0.21 -> 0.556±0.31 | Helped |
| Add Ask semantic-dictionary target text instead of random words (ablation) | AAA · Flick30k, ε=15; VIT-GPT2 | METEOR: 0.378±0.25 -> 0.556±0.31 | Helped |
| Add Attend attention heatmap to reduce the search space (ablation) | AAA · Flick30k, ε=15; Show-Attend-Tell | METEOR: 0.438±0.15 -> 0.617±0.25 | Helped |
| Add Ask semantic-dictionary target text instead of random words (ablation) | AAA · Flick30k, ε=15; Show-Attend-Tell | METEOR: 0.341±0.15 -> 0.617±0.25 | Helped |
Surprising findings
- Gray-box attacks lose most of their effectiveness when the image encoder is swapped for CLIP's encoder, with black-box adaptations performing near clean-sample level.
Limitations
- Computation time is longer than gray-box methods because no gradients are used.
- Evaluated only on two image-to-text models and Flickr30k.
Experimental setup
Datasets: Flickr30k
Benchmarks: Flickr30k
Metrics: METEOR BLEU CLIP score SPICE mean per-pixel perturbation (epsilon)
Baselines: transfer (gray) transfer+query (gray) transfer (black) transfer+query (black) AAA (w/o Attend) AAA (w/o Ask)
Builds on: differential evolution Grad-CAM CLIP
Closest papers by meaning
- A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1 (NeurIPS 2025 · similarity 0.88)
- Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks (CVPR 2025 · similarity 0.87)
- HQA-VLAttack: Towards High Quality Adversarial Attack on Vision-Language Pre-Trained Models (NeurIPS 2025 · similarity 0.87)
- UPAM: Unified Prompt Attack in Text-to-Image Generation Models Against Both Textual Filters and Visual Checkers (ICML 2024 · similarity 0.86)
- BRUSLEATTACK: A QUERY-EFFICIENT SCORE- BASED BLACK-BOX SPARSE ADVERSARIAL ATTACK (ICLR 2024 · similarity 0.86)
- Attention! Your Vision Language Model Could Be Maliciously Manipulated (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.