AI Paper Atlas

Research map / Adversarial robustness and security / Backdoor and poisoning

CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning

Hritik Bansal, Nishad Singhi, Yu Yang, Fan Yin, Aditya Grover, Kai-Wei Chang

ICCV 2023 · Adversarial robustness and security › Backdoor and poisoning · approach: Backdoors in large multimodal models · 61 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
finetune poisoned CLIP with CleanCLIP (multimodal + self-supervised loss) on clean pairspoisoned CLIP (CC3M) · ImageNet-1K val, BadNetattack success rate (%): 99.94 -> 10.46Improved
finetune with multimodal contrastive loss only (ablation)poisoned CLIP (CC3M) · ImageNet-1K val, BadNetattack success rate (%): 99.94 -> 99.8No clear effect
finetune with self-supervised loss only (ablation)poisoned CLIP (CC3M) · ImageNet-1K val, BadNetASR and clean accuracy: ASR 99.94 -> 0.9; CA 19.06 -> 13.05Mixed
add self-supervised loss during pretraining on poisoned dataCLIP pretraining · ImageNet-1K val, 4 backdoor attacksattack success rate (%): BadNet 99.94 -> 90.72; Blended 99.45 -> 99.16No clear effect
add multimodal anti-backdoor learning (ABL) unlearning during pretrainingCLIP pretraining · ImageNet-1K val, 4 backdoor attacksattack success rate (%): BadNet 99.94 -> 99.89No clear effect
apply CleanCLIP to a poisoned CLIP-400M modelCLIP RN-50 400M · ImageNet-1K val, BadNetASR and zero-shot accuracy: ASR 94.6% -> 17%; accuracy 59.6% -> 57%Improved
supervised finetuning of vision encoder on 50K labeled ImageNet imagespoisoned CLIP (CC3M) · ImageNet-1K val, four attacksattack success rate (%): BadNet 99.94 -> 0; clean accuracy 19.06 -> 40.86Improved
increase self-supervision weight lambda2 from 0.5 to 8 (ablation)CleanCLIP · Blended attackattack success rate (%)Improved
use MSCOCO or SBU Captions instead of CC3M as the clean finetuning data (ablation)CleanCLIP · 4 attacks averageASR and clean accuracy (%): ASR 95.53 -> 9.76 (MSCOCO), 6.92 (SBU); CA 18.88 -> 15.09, 15.11Mixed
increase CleanCLIP finetuning set from 10K to 100K pairs (ablation)CleanCLIP · 4 attacks averageattack success rate (%): 21.65 -> 7.21Improved
increase number of BadNet poisons from 75 to 1500 (ablation)CleanCLIP defense · BadNet, CC3Mattack success rate (%): 2.38 -> 7.7Hurt
increase pretraining data size from 500K to 3M (lower poison ratio) (ablation)CleanCLIP defense · BadNet, 1500 poisonsattack success rate (%): 24.66 -> 7.7Improved

Later papers that compared with this method

2 later accepted papers report CleanCLIP as a baseline in their results tables; 2 of them report a better number on most of the results they share with it.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy