Research map / Adversarial robustness and security / Backdoor and poisoning

BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP

CVPR 2024 · Adversarial robustness and security › Backdoor and poisoning · approach: Backdoors in large multimodal models · 24 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BadCLIP, a backdoor attack that injects a learnable image trigger and a trigger-aware context generator into CLIP during prompt learning, requiring only few-shot downstream data.

Problem

Existing backdoor attacks on CLIP require fine-tuning the whole model with large additional data or pre-training data poisoning, which is infeasible in data-limited few-shot prompt-learning settings.

Approach

Influence both image and text encoders with the trigger: a learnable additive image trigger plus a context generator that conditions learned prompts on the image (trigger-aware), so backdoor images shift text features toward the target class; a trigger warm-up stage stabilizes joint optimization.

Key results

• On seen classes across 11 datasets, BadCLIP averages 79.55% ACC (vs CoOp 80.47 and CoCoOp 79.55 range) with 99.52% ASR; unseen-class ASR averages 99.02%.
• Cross-dataset transfer from ImageNet to 10 datasets yields 100% ASR on 9 of 10 datasets with average ACC 65.31%.
• BadCLIP outperforms BadEncoder on STL10 with 95.13% ACC and 98.57% ASR using no additional data, while BadEncoder drops to 17.39% ASR with 1,000 samples.

Measured effects reported in the paper

Surprising findings

• Trigger-agnostic prompts achieve 95.31% seen-class ASR but only 2.21% unseen-class ASR, showing that trigger-aware prompts are essential for generalization.

Limitations

• Attack requires access to CLIP weights and a small downstream dataset for prompt learning (threat model is a malicious prompt-learning provider).
• CLP defense can only remove the backdoor at pruning ratios that significantly reduce clean accuracy.

Experimental setup

Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 STL10

Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNetV2

Metrics: ACC ASR harmonic mean PSNR SSIM R@1

Baselines: CLIP zero-shot CoOp CoCoOp BadEncoder fine-tuning on poisoned data trigger-agnostic prompts

Builds on: CLIP CoOp CoCoOp BadEncoder Neural Cleanse

Closest papers by meaning

• BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning (CVPR 2024 · similarity 0.90)
• Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attacks (ICML 2024 · similarity 0.89)
• Detecting Backdoor Samples in Contrastive Language Image Pretraining (ICLR 2025 · similarity 0.88)
• Test-Time Multimodal Backdoor Detection by Contrastive Prompting (ICML 2025 · similarity 0.88)
• ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-training (NeurIPS 2025 · similarity 0.87)
• Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning (NeurIPS 2025 · similarity 0.87)