Research map / Adversarial robustness and security / Backdoor and poisoning
Test-Time Multimodal Backdoor Detection by Contrastive Prompting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BDetCLIP, a training-free test-time backdoor detector for CLIP that uses GPT-4-generated benign and malignant class prompts and the contrastive similarity difference between them to flag backdoored images.
Problem
Backdoored CLIP models produce tampered image embeddings for triggered inputs; existing defenses act at pre-training or fine-tuning and need costly parameter updates or white-box access, so a cheap, black-box inference-time detector is needed.
Approach
Backdoored image embeddings are insensitive to changes in class description texts. BDetCLIP generates benign class descriptions (GPT-4) and malignant class-plus-random-sentence prompts, then uses the summed difference in cosine similarity between each image and the two prompt types as a detection score, flagging low-difference images as backdoored.
Key results
- Average AUROC of 0.938 on multimodal attacks and 0.972 on traditional attacks, vs. 0.934 for the best baseline TeCo on traditional attacks.
- Inference time on ImageNet-1K of 3m 8s, versus 253m for STRIP and 637m for TeCo.
- Defense against CleanCLIP-trained models reduces ASR for BadNet to 0.0011 and Blended to 0.0003, vs. 0.1046 and 0.0980 for CleanCLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BDetCLIP instead of the strongest unimodal baseline, SCALE-UP | CLIP test-time backdoor detection on ImageNet-1K multimodal attacks · ImageNet-1K, multimodal attacks (average) | AUROC: 0.771 -> 0.938 | Helped |
| Use BDetCLIP instead of the strongest baseline, TeCo | CLIP test-time backdoor detection on ImageNet-1K traditional attacks · ImageNet-1K, traditional attacks (average) | AUROC: 0.934 -> 0.972 | Helped |
| Use BDetCLIP instead of TeCo | CLIP backdoor detection on Food-101 · Food-101, BadNet and Blended (average) | AUROC: 0.892 -> 0.959 | Helped |
| Use BDetCLIP instead of TeCo | CLIP backdoor detection on Caltech-101 · Caltech-101, BadNet and Blended (average) | AUROC: 0.775 -> 0.983 | Helped |
| Use 7 rather than 1 class-specific benign prompt (ablation) | BDetCLIP · ImageNet-1K, BadNet/Blended/BadNet-LC/Blended-LC | AUROC (average): 0.952 -> 0.980 | Helped |
| Use 7 rather than 3 class-specific benign prompts (ablation) | BDetCLIP · ImageNet-1K, BadNet/Blended/BadNet-LC/Blended-LC | AUROC (average): 0.972 -> 0.980 | Helped |
| Use 7 rather than 5 class-specific benign prompts (ablation) | BDetCLIP · ImageNet-1K, BadNet/Blended/BadNet-LC/Blended-LC | AUROC (average): 0.977 -> 0.980 | Helped |
| Allow malignant random sentences up to 20 rather than 10 words (ablation) | BDetCLIP · ImageNet-1K, BadNet/Blended/BadNet-LC/Blended-LC | AUROC (average): 0.979 -> 0.970 | Hurt |
| Allow malignant random sentences up to 30 rather than 10 words (ablation) | BDetCLIP · ImageNet-1K, BadNet/Blended/BadNet-LC/Blended-LC | AUROC (average): 0.979 -> 0.960 | Hurt |
| Replace GPT-4 prompt generation with open-source LLM prompt generation | BDetCLIP · ImageNet-1K, BadNet and Blended | AUROC: GPT-4: 0.972, 0.983; Llama3-8B-Instruct: 0.947, 0.983; Mistral-7B-Instruct-v0.2: 0.983, 0.963 | Mixed |
| Use BDetCLIP rather than CleanCLIP defense | Backdoor defense evaluated on CleanCLIP's victim model · BadNet attack | ASR: 0.1046 -> 0.0011 ± 0.0003 | Helped |
| Use BDetCLIP rather than CleanCLIP | Backdoor defense on BadCLIP · BadCLIP | ASR: 0.9196 -> 0.0444 ± 0.0110 | Helped |
Surprising findings
- STRIP performs near or below chance (AUROC 0.025 on Blended) on ImageNet in some settings, showing unimodal detectors transfer poorly to CLIP.
- Open-source LLMs (e.g., Llama3-8B, Mistral-7B) can replace GPT-4 for prompt generation with comparable detection AUROC.
Limitations
- Only CLIP is studied as the victim model.
- Threshold selection uses a simple percentile rule on a small clean validation sample and is acknowledged as relatively simple.
Experimental setup
Datasets: ImageNet-1K Food-101 Caltech-101 CC3M
Benchmarks: ImageNet-1K Food-101 Caltech-101 STL-10
Metrics: AUROC accuracy recall F1 inference time ASR
Baselines: STRIP SCALE-UP TeCo CleanCLIP CleanerCLIP PAR RoCLIP
Builds on: CLIP CleanCLIP SCALE-UP TeCo STRIP
Closest papers by meaning
- BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning (CVPR 2024 · similarity 0.90)
- Detecting Backdoor Samples in Contrastive Language Image Pretraining (ICLR 2025 · similarity 0.90)
- Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning (NeurIPS 2025 · similarity 0.88)
- BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP (CVPR 2024 · similarity 0.88)
- ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-training (NeurIPS 2025 · similarity 0.87)
- A Closer Look at Backdoor Attacks on CLIP (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.