Research map / Vision-language and multimodal / Cross-modal learning
SEER: Backdoor Detection for Vision-Language Models through Searching Target Text and Image Trigger Jointly
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEER, a backdoor detection method for CLIP-style vision-language models that jointly searches for an image trigger and a target text in the shared feature space, flagging a model as backdoored when an anomaly index is high.
Problem
Detecting backdoors in pre-trained vision-language models when the defender lacks training data, downstream task knowledge, and the target text, which makes enumerating candidate target texts infeasible.
Approach
Initialize the target text feature as the mean of all dictionary text features and jointly optimize an image trigger (mask and pattern) and the target text feature to maximize image-text cosine similarity over clean images; the resulting text ranking and an anomaly index (-log(1-S)) reveal backdoors without enumerating texts.
Key results
- SEER achieves 10/10 detection on BadNet and Blended attacks and 8/10 on Dynamic and BadEncoder, with 0/10 false positives across architectures (over 92% detection rate claimed).
- Backdoored models yield anomaly index values above the threshold of 3 (e.g., 3.86-4.34) while clean models stay below 3 (e.g., 1.65-2.63).
- Existing fine-tuning-based defenses (Fine-Tuning, Fine-Pruning, NAD) fail to remove backdoors, with ASR remaining high (e.g., 90-99% on several attacks).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Joint search of image trigger and target text (SEER) vs. fine-tuning-based defenses | CLIP backdoored models · BadNet/Blended/Dynamic/BadEncoder attacks | detection rate (DSR): SEER DSR 8/10 to 10/10 across attacks; baselines retain high ASR | Helped |
| Fine-tuning defense applied to backdoored CLIP image encoder | Fine-Tuning baseline · BadNet RN101, Oxford Pet | ASR: ASR stays at 96.84 (backdoor not removed) | Hurt |
| Fine-pruning defense applied to backdoored CLIP image encoder | Fine-Pruning baseline · BadEncoder RN50, GTSRB-related downstream | ASR: ASR 99.72 after pruning, versus 99.83 backdoored | No clear effect |
| Trigger size increased from 4x4 to 32x32 on backdoored ViT-B/16 (ablation) | SEER search · ViT-B/16, basketball target | AI: AI stays above threshold 3 (4.2 at 4x4, 3.53 at 32x32); top text basketball in most cases | No clear effect |
| Multiple target texts and triggers (1 to 8) injected in one model (ablation) | SEER search on ViT-B/16 · multi-target backdoor | AI: AI 4.2 (1 target), 4.48 (2), 4.83 (4), 4.0 (8); all above threshold 3 | No clear effect |
| Unusual target keywords and multi-word phrases (symbol, sentiment word, phrases) (ablation) | SEER search on ViT-B/32 · backdoored target text variants | AI: AI 3.30 to 4.43; all above threshold 3 | No clear effect |
| Random initialization of target text feature instead of mean dictionary feature (ablation) | SEER search · backdoor searching | effectiveness/stability | Hurt |
| Joint search with mean-text initialization of target feature (SEER) | SEER search · backdoor searching on CLIP | TSR: TSR 10/10 on most attacks, 7/10 on BadEncoder | Helped |
Surprising findings
- Random initialization of the target text often leads to local minima, while mean-dictionary initialization substantially improves effectiveness and stability.
- Models with more injected triggers/target texts were easier to detect, as more convergence directions exist in the joint feature space.
Limitations
- Detection rate is below 100% for some attacks (8/10 for Dynamic and BadEncoder), and the claimed 92% average is not uniform.
- Evaluated only on CLIP-style dual-encoder models with four attack types and a limited number of models per architecture.
- Searching is restricted to a fixed BPE dictionary of 49,152 tokens, so target texts outside this vocabulary are not covered.
Experimental setup
Datasets: MSCOCO Flickr30k STL10 Oxford-IIIT Pet ImageNet Flickr8k GTSRB
Benchmarks: Oxford-IIIT Pet ImageNet MSCOCO Flickr8k GTSRB STL10
Metrics: Detection Success Rate False Positive Rate Text Success Rate Anomaly Index Clean Accuracy Attack Success Rate
Baselines: Fine-Tuning Fine-Pruning NAD Neural Cleanse TABOR ABS
Builds on: Neural Cleanse CLIP BadNet BadEncoder
Closest papers by meaning
- Backdooring Vision-Language Models with Out-Of-Distribution Data (ICLR 2025 · similarity 0.90)
- Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift (CVPR 2025 · similarity 0.88)
- Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignment (CVPR 2024 · similarity 0.87)
- Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Models (CVPR 2025 · similarity 0.86)
- Everyday Object Meets Vision-and-Language Navigation Agent via Backdoor (NeurIPS 2024 · similarity 0.86)
- Test-Time Multimodal Backdoor Detection by Contrastive Prompting (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.