Research map / Adversarial robustness and security / Attacks on text and graphs
AdvEDM: Fine-grained Adversarial Attack against VLM-based Embodied Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdvEDM, a gray-box fine-grained adversarial attack that removes or adds a single object's semantics in an image via vision-text encoder embeddings, so VLM-based embodied decision-making outputs valid but incorrect decisions.
Problem
Gray-box adversarial attacks on VLMs disrupt most image semantics, causing invalid outputs that do not steer VLM-based embodied decision-making agents (autonomous driving, robotic manipulation); a fine-grained attack is needed to induce valid but incorrect decisions.
Approach
Select key patches via cosine similarity between patch token embeddings and target object text embedding, then optimize the image with [CLS] alignment, masked-patch alignment, and attention-[patch] fixation so only the target object's semantics change while other semantics are preserved. Two variants: AdvEDM-R (remove object) and AdvEDM-A (add object using a generated reference image and attention
Key results
- In general image description (MS-COCO), AdvEDM-R and AdvEDM-A reach average ASR 76.8% and 70.2% with SPR 66.7% and 71.6%, while baselines have SPR under 20%.
- On DriveLM-nuScenes, average ASR is 80.8% for AdvEDM-R and 76.3% for AdvEDM-A, versus at most 19.5% for baselines.
- In robotic arm manipulation, average ASR is 68.8% (AdvEDM-R) and 64.5% (AdvEDM-A), versus at most 12.5% for baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use fine-grained object removal with AdvEDM-R instead of MF-ii | VLM image description · MS-COCO 2014 | SPR (%): 11.3 -> 66.7 | Helped |
| Use fine-grained object addition with AdvEDM-A instead of MF-ii | VLM image description · MS-COCO 2014 | SPR (%): 11.3 -> 71.6 | Helped |
| Use AdvEDM-R rather than PGD | VLM-based autonomous driving · Dolphins Benchmark | ASR (%): 17.3 -> 74.5 | Helped |
| Use AdvEDM-A rather than PGD | VLM-based autonomous driving · DriveLM-nuScenes | ASR (%): 19.5 -> 76.3 | Helped |
| Use AdvEDM-R rather than MF-it | VLM-based robotic manipulation · Robotic arm manipulation | ASR (%): 12.5 -> 68.8 | Helped |
| Use a larger perturbation constraint than 4/255 (ablation) | AdvEDM · MS-COCO | ASR | Helped |
| Increase the perturbation constraint to 12/255 or higher (ablation) | AdvEDM · MS-COCO | SPR | Hurt |
| Add the global-object loss L_cls (ablation) | AdvEDM-R and AdvEDM-A · MS-COCO | ASR | Helped |
| Add the local-object loss L_p (ablation) | AdvEDM-R and AdvEDM-A · MS-COCO | ASR | Helped |
| Add attention-patch fixation loss L_fix (ablation) | AdvEDM-R · MS-COCO | SPR: about 0.35 | Helped |
| Add attention-patch fixation loss L_fix (ablation) | AdvEDM-A · MS-COCO | SPR: about 0.4 | Helped |
| Transfer from gray-box to black-box attacks | AdvEDM-R · MS-COCO image description | ASR (%) | Hurt |
Surprising findings
- Existing semantic-disrupting attacks produce error messages such as 'no object found' rather than wrong decisions, so higher raw ASR does not translate into effective embodied attacks.
Limitations
- Attack is digital only: it requires manipulating the uploaded image, not physically deployable perturbations.
- Black-box transfer to commercial VLMs (GPT-4o, Gemini, Claude) is weaker, with ASR of 57.7% for AdvEDM-R and 39.0% for AdvEDM-A.
- Only object presence or absence is controlled; spatial location or object relations are not addressed.
Experimental setup
Datasets: MS-COCO 2014 Dolphins Benchmark DriveLM-nuScenes
Benchmarks: MS-COCO 2014 Dolphins Benchmark DriveLM-nuScenes robotic arm manipulation (100 self-collected images)
Metrics: ASR SPR semantic similarity
Baselines: CLIP-Based PGD MF-it MF-ii
Builds on: CLIP-based PGD attack MF-it/MF-ii (feature-based attacks) chain-of-thought SSA-CWA
Closest papers by meaning
- ReaLM: A Unified Red-Teaming Benchmark for Physical-World VLMs (NeurIPS 2026 · similarity 0.88)
- BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled Optimization (NeurIPS 2025 · similarity 0.87)
- Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-Based Decision-Making Systems (ICLR 2025 · similarity 0.87)
- Attention! Your Vision Language Model Could Be Maliciously Manipulated (NeurIPS 2025 · similarity 0.87)
- Pandora's Box: Towards Building Universal Attackers against Real-World Large Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- TRAP: Hijacking VLA CoT-Reasoning via Adversarial Patches (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.