Research map / Vision-language and multimodal / Robot action and navigation
Everyday Object Meets Vision-and-Language Navigation Agent via Backdoor
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IPR Backdoor, an object-aware backdoor attack on vision-and-language navigation agents that makes them STOP when a physical object trigger (yoga ball, wall painting, door) is seen, while keeping navigation performance.
Problem
VLN agents deployed in real homes can be backdoored so that they halt at triggers placed in the physical environment; this security risk has been overlooked, and standard backdoor methods do not transfer to VLN's cross-modal, continuous decision-making setting.
Approach
Injects backdoors across imitation learning, pretraining, and RL: a See2Stop loss for IL, an Anchor loss aligning poisoned visual features to the text "Stop" plus a Consistency loss to avoid trivial solutions in pretraining, and a Backdoor-aware reward that replaces navigation reward in RL to keep the attack alive.
Key results
- With yoga ball trigger, HAMT IL+RL with IPR Backdoor reaches 100% Att-SR and 66.18% SR, close to the 65.90% SR of the clean HAMT IL+RL baseline.
- Digital triggers (black-white patch, sig) also achieve 100% Att-SR with SR of 65.01% and 63.81% for HAMT IL+RL.
- Using only the navigation reward in RL drops Att-SR by 27%, while the Backdoor-aware Reward restores Att-SR to 100%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the See2Stop loss during imitation learning (ablation) | HAMT · Matterport3D, yoga ball trigger | Att-SR: - -> 75 | Helped |
| Add the Anchor loss to the See2Stop setup (ablation) | HAMT · Matterport3D, yoga ball trigger | SR: 57.34 -> 40.05 | Hurt |
| Add the Consistency loss alongside the Anchor loss (ablation) | HAMT · Matterport3D, yoga ball trigger | Att-SR: 2 -> 100 | Helped |
| Use the navigation reward during reinforcement learning rather than the preceding backdoor-focused objective (ablation) | HAMT · Matterport3D, yoga ball trigger | Att-SR: 100 -> 73 | Hurt |
| Replace the navigation reward with the Backdoor-aware Reward (ablation) | HAMT · Matterport3D, yoga ball trigger | Att-SR: 73 -> 100 | Helped |
| Use IPR Backdoor with HAMT after IL+RL instead of the HAMTILRL baseline | HAMT · Matterport3D, yoga ball trigger | SR: 65.90 -> 66.18 | Helped |
| Use IPR Backdoor with HAMT after IL+RL instead of the HAMTILRL baseline | HAMT · Matterport3D, yoga ball trigger | Att-SR: - -> 100 | Helped |
Surprising findings
- Using only the Anchor Loss in pretraining causes a trivial solution where clean and poisoned features collapse near the anchor, yielding 40.05% SR and 2% Att-SR.
- The conventional navigation reward in RL reduces backdoor attack success by 27%, showing that RL undermines backdoors learned earlier.
Limitations
- Only explores the STOP action as the abnormal behavior.
- Attack tests use teacher-forcing navigation planning to ensure the agent encounters the trigger.
- Evaluated on a small set of triggers (three physical objects, two digital patterns) and two VLN agents.
Experimental setup
Datasets: Matterport3D R2R RxR
Benchmarks: R2R RxR
Metrics: Success Rate SPL Navigation Error Trajectory Length Attack Success Rate
Baselines: HAMT (IL) HAMT (IL+RL) RecBert (IL) RecBert (IL+RL)
Builds on: HAMT RecBert BadNets
Closest papers by meaning
- BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learning (ICLR 2026 · similarity 0.88)
- BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled Optimization (NeurIPS 2025 · similarity 0.87)
- SEER: Backdoor Detection for Vision-Language Models through Searching Target Text and Image Trigger Jointly (AAAI 2024 · similarity 0.86)
- IAG: Input-aware Backdoor Attack on VLM-based Visual Grounding (CVPR 2026 · similarity 0.86)
- Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions (NeurIPS 2024 · similarity 0.85)
- Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.