Research map / Vision-language and multimodal / CLIP and zero-shot
ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ToxicTextCLIP, a background-sensitive framework that selects and augments poisoned texts to attack CLIP pre-training via text-only poisoning and backdoors, evading RoCLIP, CleanCLIP and SafeCLIP.
Problem
Text-modality poisoning and backdoor attacks on CLIP during pre-training are underexplored; existing caption-substitution attacks produce target-inconsistent background content and lack scalable supply of background-consistent poisoned texts.
Approach
Iteratively combine a background-aware selector (removing up to eta class-relevant words to isolate background and scoring background-image alignment minus background-class alignment) with a background-driven augmenter (CLIP text feature plus scaled image feature decoded by a cross-attention Transformer with diverse beam search and Jaccard post-filtering) to generate diverse, background-consistent
Key results
- On CC3M, single-target poisoning ASR reaches 95.83% vs 62.5% for mmPoison, with clean accuracy 32.23%.
- Sentence-level backdoor Hit@1 reaches 98.68% on CC3M without defense, and 91.15% under RoCLIP.
- Under SafeCLIP on CC3M, single-target poisoning ASR is 64.17% for ToxicTextCLIP vs 25.00% for mmPoison.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ToxicTextCLIP instead of mmPoison | CLIP, CC3M STI-P · CC3M | ASR: 62.50 -> 95.83 | Helped |
| use ToxicTextCLIP instead of mmPoison | CLIP, YFCC STI-P · YFCC | ASR: 66.67 -> 91.67 | Helped |
| use ToxicTextCLIP instead of the random-text baseline | CLIP, CC3M W-BD · CC3M | Hit@1: 72.13 -> 92.66 | Helped |
| use ToxicTextCLIP instead of the random-text baseline | CLIP, CC3M S-BD · CC3M | Hit@1: 64.41 -> 98.68 | Helped |
| use ToxicTextCLIP instead of the random-text baseline | CLIP, YFCC W-BD · YFCC | Hit@1: 50.47 -> 70.62 | Helped |
| use ToxicTextCLIP instead of the random-text baseline | CLIP, YFCC S-BD · YFCC | Hit@1: 67.04 -> 79.10 | Helped |
| add background-aware selector (rather than random target texts) (ablation) | ToxicTextCLIP, CC3M STI-P · CC3M STI-P, no defense | ASR: 87.50 -> 95.83 | Helped |
| add background-driven augmenter (ablation) | ToxicTextCLIP, CC3M STI-P · CC3M STI-P, no defense | ASR: 83.33 -> 95.83 | Helped |
| add background-aware selector (rather than random target texts) (ablation) | ToxicTextCLIP, CC3M W-BD · CC3M W-BD, no defense | Hit@1: 65.26 -> 92.66 | Helped |
| add background-driven augmenter (ablation) | ToxicTextCLIP, CC3M W-BD · CC3M W-BD, no defense | Hit@1: 81.32 -> 92.66 | Helped |
| add background-aware selector (rather than random target texts) (ablation) | ToxicTextCLIP, CC3M S-BD · CC3M S-BD, no defense | Hit@1: 78.91 -> 98.68 | Helped |
| add background-driven augmenter (ablation) | ToxicTextCLIP, CC3M S-BD · CC3M S-BD, no defense | Hit@1: 91.34 -> 98.68 | Helped |
Surprising findings
- Word-level and sentence-level backdoors are effective even at low poisoning rates, with 50 to 75 poisoned samples pushing Hit@5 and Hit@10 above 80% on CC3M.
- The ONION text-only defense detects only about 3.2% of poisoned samples for the sentence-level trigger.
Limitations
- Only English-language CLIP models are studied.
- Effectiveness may vary in specialized domains such as medical or satellite imagery, or with proprietary caption corpora.
- Simple rare-word and fixed-sentence triggers are used as proof of concept; complex or natural triggers are not explored.
Experimental setup
Datasets: CC3M CC12M YFCC15M COCO ImageNet
Benchmarks: CC3M validation COCO
Metrics: clean accuracy attack success rate Hit@1 Hit@5 Hit@10 perplexity distinct n-grams
Baselines: mmPoison random-trigger caption injection baseline RoCLIP CleanCLIP SafeCLIP ONION
Builds on: CLIP mmPoison CleanCLIP RoCLIP SafeCLIP
Closest papers by meaning
- Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attacks (ICML 2024 · similarity 0.93)
- Detecting Backdoor Samples in Contrastive Language Image Pretraining (ICLR 2025 · similarity 0.89)
- BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning (CVPR 2024 · similarity 0.88)
- BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP (CVPR 2024 · similarity 0.87)
- Test-Time Multimodal Backdoor Detection by Contrastive Prompting (ICML 2025 · similarity 0.87)
- Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.