Research map / Vision-language and multimodal / Human and embodied modeling
Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Semantic Shield, a training-time defense for contrastive vision-language models that aligns image patches with LLM-generated knowledge elements and penalizes attention to poorly aligned patches, defending against backdoor and poisoning attacks.
Problem
Contrastively trained vision-language models scraped from the web are vulnerable to backdooring and poisoning attacks that inject small amounts of malicious image-caption data.
Approach
Use knowledge elements (sub-object parts, attributes) extracted from captions by an LLM to learn patch-KE alignment via multi-instance constraints, then constrain ViT attention to be proportional to patch-KE alignment and down-weight the contrastive loss for samples with low alignment.
Key results
- On COCO, Weighted CL + Attention reduces Hit@1 under backdoor patch attack from 90.66 (no defense) to 0.9, versus 5.35 for CleanCLIP.
- On COCO single-target poisoning (dog2boat), Weighted CL + Attention reduces Hit@1 from 18.0 to 0.04, versus 3.39 for CleanCLIP.
- Defended models retain utility comparable to the clean model, e.g., COCO IR Recall@10 of 74.22-75.70 versus 75.13 clean.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Semantic Shield (weighted contrastive loss plus attention guidance) | CL · COCO and Flickr30k backdoor attacks (patch, BPP, Wanet) | Hit@1: 90.66 -> 0.9; 100.0 -> 0.0; 100.0 -> 0.0; 91.97 -> 0.0; 100.0 -> 0.0; 100.0 -> 0.0 | Helped |
| Apply Semantic Shield (weighted contrastive loss plus attention guidance) | CL · COCO single-target dog2boat poisoning | Hit@1: 18.0 -> 0.04 | Helped |
| Apply Semantic Shield (weighted contrastive loss plus attention guidance) | CL · COCO multi-target dog2boat poisoning | Hit@1: 77.12 -> 2.23 | Helped |
| Apply Semantic Shield (weighted contrastive loss plus attention guidance) | CL · COCO multi-target train2zebra poisoning | Hit@1: 55.32 -> 0.0 | Helped |
| Apply Semantic Shield (weighted contrastive loss plus attention guidance) | CL · Flickr30k multi-target bird2sofa poisoning | Hit@1: 55.32 -> 0.0 | Helped |
| Add patch-KE alignment loss (ablation) | CL + Attention · COCO, Backdoor Patch | Hit@1: 4.20 -> 0.9 | Helped |
| Add patch-KE alignment loss (ablation) | CL + Attention · Flickr30k, Backdoor Patch | Hit@1: 1.20 -> 0.0 | Helped |
| Add attention guidance (ablation) | CL + KE · COCO, Backdoor Patch | Hit@1: 9.0 -> 0.9 | Helped |
| Add attention guidance (ablation) | CL + KE · Flickr30k, Backdoor Patch | Hit@1: 16.10 -> 0.0 | Helped |
| Use patch-KE alignment loss alone (ablation) | CL · COCO and Flickr30k backdoor attacks | Hit@1: CL+ KE: 9.0, 25.39, 12.21, 16.10, 13.14, 23.36 | Mixed |
| Use attention guidance without weighted contrastive loss (ablation) | CL · COCO and Flickr30k backdoor attacks | Hit@1: CL + Attention: 4.20, 0.0, 0.0, 1.20, 0.0, 0.0 | Mixed |
| Use Semantic Shield for image-caption retrieval utility | CL · COCO and Flickr30k, Recall@10 across attack settings | Recall@10: 74.22; 74.56; 74.23; 73.46; 73.51; 61.07; 56.32; 60.16; 59.76; 58.78 | No clear effect |
Surprising findings
- The unweighted CL + Attention variant slightly outperforms the full Semantic Shield on dog2boat Hit@1 in poisoning experiments.
- CL + KE alone is weaker than CleanCLIP on several backdoor settings (e.g., Wanet Hit@10 of 88.38 on COCO), showing the attention constraint is needed.
Limitations
- May be less effective on charts or abstract text where clear knowledge elements cannot be extracted.
- Does not defend against attacks on the language model via captions.
- Relies on LLM-generated knowledge elements, which are only potentially relevant to the image.
Experimental setup
Datasets: COCO Flickr30k ImageNet-21k ImageNet-1k
Benchmarks: COCO Flickr30k
Metrics: Hit@1 Hit@5 Hit@10 Recall@10
Baselines: CleanCLIP ABL (Anti-backdoor learning) CL (no defense) CL + KE CL + Attention
Builds on: CLIP CleanCLIP Anti-backdoor learning Carlini & Terzis poisoning attack
Closest papers by meaning
- Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift (CVPR 2025 · similarity 0.90)
- Backdooring Vision-Language Models with Out-Of-Distribution Data (ICLR 2025 · similarity 0.90)
- FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (AAAI 2025 · similarity 0.89)
- SLADE: Shielding against Dual Exploits in Large Vision-Language Models (CVPR 2025 · similarity 0.88)
- Semantic Representation Attack against Aligned Large Language Models (NeurIPS 2025 · similarity 0.88)
- BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.