Research map / Adversarial robustness and security / Backdoor and poisoning
Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Repulsive Visual Prompt Tuning (RVPT), which tunes deep visual prompts on few-shot clean data with a feature-repelling loss so a backdoored CLIP encodes only in-dataset predictive features, defending against multimodal backdoor attacks.
Problem
Backdoor attacks on contrastively pre-trained CLIP models persist into downstream tasks; existing defenses either fine-tune the whole model or need poisoned data, which is costly or unrealistic when only few-shot clean downstream data is available.
Approach
CLIP's low perturbation resistivity stems from encoding off-dataset features; deep visual prompt tuning with a feature-repelling loss (minimizing cosine similarity between prompted and original deep-layer features) plus cross-entropy forces the encoder to keep only predictive downstream features, suppressing trigger features without poisoned data.
Key results
- On ImageNet, RVPT reduces ASR of BadCLIP from 99.83% (no defense) and 89.70% (CleanCLIP) to 2.76% with clean accuracy 61.81%.
- RVPT tunes only about 0.27% of CLIP parameters (0.34M) and reaches ASR of 0.05% on BadNet and 0.02% on Blended on ImageNet.
- RVPT generalizes across datasets: it keeps ASR low on Caltech101 and OxfordPets when tuned on ImageNet, and on text-to-image retrieval reduces ASR from 74.67% to 0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add RVPT instead of CleanCLIP | backdoored CLIP ViT/B32 · ImageNet1K, BadCLIP | ASR: 89.70 -> 2.76 | Helped |
| add RVPT instead of Linear Probe | backdoored CLIP ViT/B32 · ImageNet1K, BadCLIP | ASR: 99.70 -> 2.76 | Helped |
| use RVPT rather than VPT alone | backdoored CLIP in image captioning · Image captioning, BadCLIP | ASR: 81.21 -> 0 | Helped |
| use RVPT rather than VPT alone | backdoored CLIP in text-to-image retrieval · Text-to-image retrieval, BadCLIP | ASR: 22.67 -> 0 | Helped |
| add feature-repelling loss to few-shot visual prompt tuning (ablation) | CLIP tuned on few-shot ImageNet · ImageNet, BadCLIP | ASR: 98.14 -> 2.76 | Helped |
| add feature-repelling loss to full-model CE fine-tuning (ablation) | CLIP fine-tuned on ImageNet · ImageNet, BadCLIP | ASR: 10.09 -> 0 | Helped |
| add feature-repelling loss to CC3M Info-NCE fine-tuning (ablation) | CLIP fine-tuned on CC3M · ImageNet, BadCLIP | ASR: 92.50 -> 18.23 | Helped |
| increase the number of shots per class (ablation) | RVPT · ImageNet, BadCLIP and other attacks | ASR | Helped |
| increase balancing factor α (ablation) | RVPT · evaluated datasets | ASR | Mixed |
| increase visual context length beyond approximately 25 to 50 tokens (ablation) | RVPT · evaluated attacks | CA | Hurt |
| tune with a visual context length of 5 instead of the default 50 (ablation) | RVPT · evaluated attacks | ASR | Helped |
| embed prompts in later layers rather than earlier layers (ablation) | RVPT · evaluated attacks | CA | Helped |
Surprising findings
- Even a non-backdoored CLIP shows a significant drop in poisoned accuracy, suggesting low visual feature resistivity is intrinsic to CLIP.
- Only 1 shot per class already drives ASR of Blended and BadNet near zero on ImageNet.
Limitations
- Clean accuracy drops slightly under cross-dataset transfer (ImageNet to Caltech101/OxfordPets) and under strong domain shift (ImageNet-A).
- Requires a small set of clean labeled downstream samples; not designed for adversarial examples (adversarial accuracy 4.10% vs 4.37% no defense).
- May not defend against novel attacks that do not use traditional triggers.
Experimental setup
Datasets: ImageNet1K Caltech101 OxfordPets CC3M ImageNet-V2 ImageNet-A ImageNet-R ImageNet-Sketch MSCOCO
Benchmarks: ImageNet1K Caltech101 OxfordPets ImageNet-V2 ImageNet-A ImageNet-R ImageNet-Sketch MSCOCO
Metrics: ASR clean accuracy poisoned accuracy perturbation resistivity
Baselines: No defense CleanCLIP Linear Probe VPT Adversarial Prompt Tuning
Builds on: VPT CleanCLIP CoOp BadCLIP
Closest papers by meaning
- BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning (CVPR 2024 · similarity 0.91)
- Test-Time Multimodal Backdoor Detection by Contrastive Prompting (ICML 2025 · similarity 0.88)
- SLADE: Shielding against Dual Exploits in Large Vision-Language Models (CVPR 2025 · similarity 0.87)
- Attention-Imperceptible Backdoor Attacks on Vision Transformers (AAAI 2025 · similarity 0.87)
- R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning (CVPR 2025 · similarity 0.87)
- TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.