Research map / Vision-language and multimodal / Multimodal LLMs
TRAP: Targeted Redirecting of Agentic Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TRAP, a black-box diffusion and CLIP-embedding optimization attack that makes natural-looking edited images win multi-candidate VLM selection, evaluated on COCO against SPSA, Bandit, and SA_AET baselines.
Problem
Vision-language models used in agentic selection pipelines choose among candidate images via cross-modal similarity, opening a stealthy attack surface. Existing attacks need pixel-level perturbations or privileged access to model or environment internals.
Approach
Optimizes the CLIP image embedding toward a positive attacker prompt, with a Siamese network that preserves distinctive features, a segmentation-refined layout mask, and LPIPS realism loss. The embedding is then decoded into an image through Stable Diffusion img2img, under a black-box threat model.
Key results
- TRAP reaches 100% ASR on LLaVA-1.5-34B, Gemma3-8B and Mistral-small-3.1-24B, versus at most 36% for SPSA and 6% for Bandit.
- On GPT-4o, TRAP reaches 63% ASR, compared with 12% for SA_AET and 1% for SPSA.
- Under the e-commerce webpage scenario, TRAP reaches 51% on LLaVA-1.5-34B and 50% on Gemma3-8B, the highest among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TRAP instead of the strongest listed baseline, SA_AET | LLaVA-1.5-34B · COCO | attack success rate: 85% -> 100% | Helped |
| Use TRAP instead of the strongest listed baseline, SA_AET | Gemma3-8B · COCO | attack success rate: 67% -> 100% | Helped |
| Use TRAP instead of the strongest listed baseline, SA_AET | Mistral-small-3.1-24B · COCO | attack success rate: 61% -> 100% | Helped |
| Use TRAP instead of the strongest listed baseline, SA_AET | Mistral-small-3.2-24B · COCO | attack success rate: 55% -> 99% | Helped |
| Use TRAP instead of the strongest listed baseline, SA_AET | GPT-4o · COCO | attack success rate: 12% -> 63% | Helped |
| Use TRAP instead of the strongest listed baseline, SA_AET | CogVLM · COCO | attack success rate: 42% -> 94% | Helped |
| Increase distinctive loss weight from 0.3 to 0.8 (ablation) | TRAP · COCO, four evaluated models | attack success rate: 88%, 70%, 72%, 65% | Hurt |
| Remove semantic loss (weight 0.5 to 0.0) (ablation) | TRAP · COCO, four evaluated models | attack success rate: 90%, 82%, 77%, 70% | Hurt |
| Increase perceptual loss weight from 1.0 to 1.5 (ablation) | TRAP · COCO, four evaluated models | attack success rate: 100%, 100%, 100%, 98% | No clear effect |
| Increase semantic loss weight from 0.5 to 1.0 (ablation) | TRAP · COCO, four evaluated models | attack success rate: 100%, 100%, 94%, 92% | Mixed |
| Decrease perceptual loss weight from 1.0 to 0.8 (ablation) | TRAP · COCO, four evaluated models | attack success rate: 100%, 100%, 98%, 94% | No clear effect |
| Remove distinctive loss (weight 0.3 to 0.0) (ablation) | TRAP · COCO, four evaluated models | attack success rate: 100%, 100%, 91%, 88% | Mixed |
Surprising findings
- Adding Gaussian noise to adversarial images has negligible effect on ASR, while the attack still reaches 92% on Robust-LLaVA.
Limitations
- Assumes the agent relies on contrastive vision-language similarity, which may not hold for future non-contrastive or defended systems.
- Iterative optimization and diffusion decoding make TRAP more computationally expensive than pixel-level attacks, limiting real-time use.
- Performance depends on auxiliary components such as the layout mask and diffusion model, and may degrade on edge cases.
Experimental setup
Datasets: COCO Captions Flickr8k_sketch ArtCap
Benchmarks: COCO Captions Flickr8k_sketch ArtCap
Metrics: attack success rate selection probability
Baselines: SPSA Bandit Stable Diffusion (no optimization) SSA_CWA SA_AET
Builds on: Stable Diffusion CLIP LPIPS SPSA Bandit
Closest papers by meaning
- MaskSense: Confronting the Visual Exploration Trap in Masked Image Generation (NeurIPS 2026 · similarity 0.86)
- AdvEDM: Fine-grained Adversarial Attack against VLM-based Embodied Agents (NeurIPS 2025 · similarity 0.86)
- SenseSearch: Empowering Vision-Language Models with High-Resolution Agentic Search-Reasoning via Reinforcement Learning (CVPR 2026 · similarity 0.85)
- NoRD: A Data-Efficient Vision-Language-Action Model that Drives without Reasoning (CVPR 2026 · similarity 0.85)
- RAT: Adversarial Attacks on Deep Reinforcement Agents for Targeted Behaviors (AAAI 2025 · similarity 0.84)
- VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.