Research map / Adversarial robustness and security / Backdoor and poisoning

Not All Prompts Are Secure: A Switchable Backdoor Attack Against Pre-trained Vision Transfomers

CVPR 2024 · Adversarial robustness and security › Backdoor and poisoning · approach: Backdoors in large multimodal models · 9 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SWARM, a switchable backdoor attack on visual prompt tuning of pre-trained ViTs where an extra switch token toggles between a clean mode and a backdoor mode, achieving ~97.9% average ASR on VTAB-1k.

Problem

Visual prompting (prompt tuning of frozen pre-trained vision transformers) exposes a security risk from malicious cloud providers, but backdoor threats to visual prompts are unexplored; backdoors must also stay hidden in benign mode and resist detection and mitigation.

Approach

Add a single learnable switch token to the visual prompts: without it the model behaves cleanly even on triggered inputs, with it a trigger forces a target class. Training alternates a clean loss (clean prompts and trigger), a backdoor loss (switch token and trigger, clean prompts frozen), and a cross-mode feature distillation loss that aligns features with and without the switch token.

Key results

• SWARM reaches 97.90% average ASR across VTAB-1k tasks, the highest among compared backdoor attacks, with 96.32%-ish... (SWARM-B average BA 59.95 vs. 61.48 no-attack)
• In clean mode (SWARM-C), average BA 61.50 and BA-T 60.82, showing triggered and clean accuracy are nearly equal.
• Against Scale-Up detection, SWARM achieves average AUROC 0.4905 and ASR-D 61.75%, the lowest AUROC and highest ASR-D among compared attacks.

Measured effects reported in the paper

Surprising findings

• Without the cross-mode feature distillation loss, the switch-on model's benign accuracy drops by about 10% on CIFAR100 and related tasks.

Limitations

• Evaluated only on ViT-family backbones pre-trained on ImageNet-21K and on VTAB-1k tasks.
• Detection and mitigation evaluated only with a limited set of defenses (Scale-Up, TeCo, STRIP, NAD, I-BAU).

Experimental setup

Datasets: VTAB-1k ImageNet-21K

Benchmarks: VTAB-1k CIFAR-100 Caltech101 EuroSAT DMLab

Metrics: benign accuracy attack success rate AUROC ASR-D

Baselines: BadNets Blended WaNet ISSBA

Builds on: Visual Prompt Tuning (VPT) BadNets Blended WaNet ISSBA

Closest papers by meaning

• Attention-Imperceptible Backdoor Attacks on Vision Transformers (AAAI 2025 · similarity 0.88)
• Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift (CVPR 2025 · similarity 0.87)
• Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Models (CVPR 2025 · similarity 0.86)
• BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP (CVPR 2024 · similarity 0.86)
• BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled Optimization (NeurIPS 2025 · similarity 0.86)
• LOTUS: Evasive and Resilient Backdoor Attacks through Sub-Partitioning (CVPR 2024 · similarity 0.86)