Research map / Adversarial robustness and security / Backdoor and poisoning
A Closer Look at Backdoor Attacks on CLIP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical study decomposing CLIP image representations into patch, attention-head and MLP contributions to show how backdoor attacks infect components, then proposes head-selective representation repair and head-count-based backdoor sample detection at inference.
Problem
Backdoor attacks on CLIP produce tampered image representations, yet there is limited systematic understanding of which model components they affect and how, which hinders defense design.
Approach
Use representation decomposing and mean-ablation over attention heads and MLPs to show local patch triggers infect late attention heads while global perturbation triggers infect late MLPs; then repair only heavily infected heads (low cosine similarity to head prototypes) or flag samples with many infected heads at inference.
Key results
- Decomp-Rep reduces BadCLIP ASR on ImageNet-1K from 99.78% to 0.94% with CACC 56.08% (vs 60.73% no defense).
- Decomp-Det average AUROC of 0.944 across attacks, exceeding STRIP, SCALE-UP and TeCo by 0.118, 0.220 and 0.187.
- Decomp-Rep applied on top of CleanCLIP reduces BadNet ASR from 54.23% to 41.49% and BadCLIP ASR from 69.03% to 37.09% on ImageNet-1K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Repair selected infected AH representations using head-specific prototypes | CLIP ViT-B/32 backdoored with BadCLIP · ImageNet-1K | ASR: 99.78 -> 0.94 | Helped |
| Repair selected infected AH representations using head-specific prototypes | CLIP ViT-B/32 backdoored with BadNet · ImageNet-1K | ASR: 86.09 -> 21.45 | Helped |
| Repair selected infected AH representations after CleanCLIP fine-tuning | CleanCLIP · ImageNet-1K, BadCLIP | ASR: 69.03 -> 37.09 | Helped |
| Detect backdoor samples by counting selected infected AHs | CLIP ViT-B/32 · ImageNet-1K, Caltech-101, Oxford Pets (average) | AUROC: 0.826 -> 0.944 | Helped |
| Mean-ablate all MLPs (ablation) | CLIP ViT-B/32 with local patch attacks (BadNet, BadCLIP) · ImageNet-1K | ASR | No clear effect |
| Mean-ablate all MLPs (ablation) | CLIP ViT-B/32 with global perturbation attacks (Blended, ISSBA) · ImageNet-1K | ASR | Helped |
| Mean-ablate all AHs (ablation) | CLIP ViT-B/32 with local patch attacks (BadNet, BadCLIP) · ImageNet-1K | ASR | Helped |
| Mean-ablate all AHs (ablation) | CLIP ViT-B/32 with global perturbation attacks (Blended, ISSBA) · ImageNet-1K | ASR | No clear effect |
| Ablate AHs in the last layer (ablation) | CLIP ViT-B/32 with BadNet and BadCLIP · ImageNet-1K | ASR | Helped |
| Ablate MLPs in the last five layers (ablation) | CLIP ViT-B/32 with Blended and ISSBA · ImageNet-1K | ASR | Helped |
| Ablate one layer’s MLP instead of the last five layers’ MLPs (ablation) | CLIP ViT-B/32 with Blended and ISSBA · ImageNet-1K | ASR | No clear effect |
| Use fixed AH locations [1, 2, 3], [7, 8, 9], or [10, 11, 12] rather than selecting infected AHs per image (ablation) | Decomp-Rep · ImageNet-1K, Caltech-101, Oxford Pets | ASR | Hurt |
Surprising findings
- Basic representation decomposition without repair does not reduce ASR and sometimes increases it (e.g., BadNet ASR 86.09% to 88.58%), suggesting backdoor effects are mainly direct rather than indirect.
- Some last-layer attention heads remain uninfected and keep property-specific roles such as color and location.
Limitations
- Defense evaluated only against five attacks and three downstream datasets with ViT-B/32; adaptive attacks not evaluated (acknowledged as future work in impact statement).
- Performance depends on threshold epsilon: higher values reduce ASR but sharply reduce CACC.
- Backdoored CLIP is produced by fine-tuning the open CLIP model on 1,500 poisoned pairs rather than pretraining from scratch due to limited resources.
Experimental setup
Datasets: CC3M ImageNet-1K Caltech-101 Oxford Pets
Benchmarks: ImageNet-1K Caltech-101 Oxford Pets
Metrics: ASR CACC AUROC
Baselines: CleanCLIP STRIP SCALE-UP TeCo Base-Decomp
Builds on: TextSpan (Gandelsman et al., 2024) CleanCLIP representation decomposition
Closest papers by meaning
- Detecting Backdoor Samples in Contrastive Language Image Pretraining (ICLR 2025 · similarity 0.88)
- Test-Time Multimodal Backdoor Detection by Contrastive Prompting (ICML 2025 · similarity 0.86)
- Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift (CVPR 2025 · similarity 0.86)
- Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning (NeurIPS 2025 · similarity 0.86)
- BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning (CVPR 2024 · similarity 0.86)
- TCAP: Tri-Component Attention Profiling for Unsupervised Backdoor Detection in MLLM Fine-Tuning (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.