Research map / Adversarial robustness and security / Attacks on text and graphs
SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SceneTAP, a training-free GPT-4o-driven planner that generates scene-coherent typographic adversarial text, its placement, and TextDiffuser prompts to attack LVLMs in digital and physical settings.
Problem
Typographic attacks on large vision-language models use predefined adversarial text at fixed locations (center or margin) that is visually anomalous and cannot be realistically deployed in the physical world.
Approach
An LLM agent (GPT-4o) with chain-of-thought, set-of-mark segmentation prompts, and a revisable prompt plans question- and image-adaptive adversarial text, its scene-contextual placement, and a caption for a scene-coherent TextDiffuser to insert the text naturally, which can then be printed and placed physically.
Key results
- On two-choice TypoD-base questions, SceneTAP reaches 44.32% average ASR versus 29.12% for Center Attack and 26.45% for Margin Attack.
- On VQAv2, SceneTAP raises average ASR from 47.19% (no attack) to 62.10%, while Center and Margin Attacks barely change it.
- Against ChatGPT-4o, SceneTAP averages 33.79% ASR versus 15.83% without attack, 23.44% for Center Attack, and 19.05% for Margin Attack.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SceneTAP instead of Center Attack | ChatGPT-4o · TypoD-base, Visual Attribute Detection | ASR: 10.76 -> 14.87 | Helped |
| Use SceneTAP instead of Center Attack | LLaVA · VQAv2 | ASR: 68.3 -> 80 | Helped |
| Use SceneTAP instead of Center Attack | MiniGPT-v2 · LingoQA | ASR: 36.8 -> 51.8 | Helped |
| Use SceneTAP instead of Margin Attack | InstructBLIP · TypoD-base, Visual Attribute Detection | ASR: 27.69 -> 62.56 | Helped |
| Generate adversarial text strategically rather than use the fixed incorrect option (ablation) | SceneTAP on LLaVA · VQAv2 | ASR | Helped |
| Place adversarial text in contextually relevant regions rather than at the center (ablation) | SceneTAP on LLaVA · TypoD-base, Visual Attribute Detection | ASR | Helped |
| Refine placement for naturalness (ablation) | SceneTAP on LLaVA · TypoD-base, Visual Attribute Detection | ASR | Hurt |
| Insert text using diffusion techniques after placement refinement (ablation) | SceneTAP on LLaVA · TypoD-base, Visual Attribute Detection | ASR | Helped |
Surprising findings
- Center and Margin typographic attacks have almost no effect on open-ended VQAv2 accuracy (47.19% to about 47.9%), while scene-planned placement raises it substantially.
Limitations
- Relies on proprietary GPT-4o as planner and ChatGPT as the naturalness judge, so reproducibility depends on API access.
- Physical evaluation is limited to four illustrative cases rather than a systematic physical study.
- Naturalness is scored by an LLM judge rather than by human raters.
Experimental setup
Datasets: TypoD-base LingoQA VQAv2
Benchmarks: TypoD-base LingoQA VQAv2
Metrics: Attack Success Rate Naturalness Score Comprehensive Score
Baselines: Center Attack Margin Attack No Attack
Builds on: TextDiffuser set-of-mark prompting Center Attack Margin Attack
Closest papers by meaning
- MAGIC: Mastering Physical Adversarial Generation in Context Through Collaborative LLM Agents (AAAI 2026 · similarity 0.87)
- ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.87)
- Visual Adversarial Examples Jailbreak Aligned Large Language Models (AAAI 2024 · similarity 0.87)
- FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (AAAI 2025 · similarity 0.87)
- Pandora's Box: Towards Building Universal Attackers against Real-World Large Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- Stage-wise Attention-Guided Region Sequencing for Adversarial Attacks on Large Vision-Language Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.