Research map / Adversarial robustness and security / Attacks on text and graphs
TrojVLM: Backdoor Attack Against Vision Language Models
Weimin Lyu, Lu Pang, Tengfei Ma, Haibin Ling, Chao Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add semantic preservation loss to language modeling loss (ablation) | TrojVLM · Flickr8k captioning (poisoned images, sentence target) | CIDEr: 108.4 -> 114.3 | Improved |
| add semantic preservation loss to language modeling loss (ablation) | TrojVLM · Flickr8k captioning (poisoned images, word target) | ASR: 0.961 -> 0.976 | Improved |
| add semantic preservation loss to language modeling loss (ablation) | TrojVLM · OK-VQA (poisoned images, word target) | VQA score: 42.7 -> 43.7 | Improved |
| use TrojVLM instead of Blended baseline for text quality | TrojVLM · Flickr8k captioning (poisoned images) | CIDEr: 4.5 -> 114.3 | Improved |
| use TrojVLM instead of AnyDoor on poisoned images | TrojVLM · OK-VQA (poisoned images) | VQA score: 41.9 -> 45.7 | Improved |
| BadEncoder (CLIP encoder attack) applied to VLM captioning | BadEncoder · Flickr8k captioning (poisoned images) | ASR: 0.000 | Hurt |
| increase trigger size from 5 to 30 pixels (ablation) | TrojVLM · Flickr8k captioning | ASR: 0.664 -> 1.000 | Improved |
| increase poison rate from 0.05 to 0.1 (ablation) | TrojVLM · Flickr8k captioning | CIDEr: 109.1 -> 114.3 | Improved |
| increase poison rate from 0.1 to 0.3 (ablation) | TrojVLM · Flickr8k captioning | CIDEr: 114.3 -> 114.7 | No clear effect |
| change trigger location from upper left to other positions (ablation) | TrojVLM · Flickr8k captioning | ASR: 0.979 -> 0.999 | No clear effect |
| keep only the 1% of image tokens containing the trigger (ablation) | TrojVLM · Flickr8k captioning | ASR: 0.96 (word), 0.97 (sent), 0.90 (web) | No clear effect |
| apply TrojVLM to MiniGPT-4 and InstructBLIP | TrojVLM · Flickr8k captioning | ASR: 0.959 to 0.986 | Improved |
Later papers that compared with this method
1 later accepted paper reports TrojVLM as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models (ICML 2026 · better on 1 of 1 shared result)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Backdooring Vision-Language Models with Out-Of-Distribution Data (ICLR 2025 · similarity 0.93)
- Attention! Your Vision Language Model Could Be Maliciously Manipulated (NeurIPS 2025 · similarity 0.89)
- FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (AAAI 2025 · similarity 0.88)
- BadToken: Token-level Backdoor Attacks to Multi-modal Large Language Models (CVPR 2025 · similarity 0.88)
- BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models (NeurIPS 2025 · similarity 0.87)
- MMJ-Bench: A Comprehensive Study on Jailbreak Attacks and Defenses for Vision Language Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
