Research map / Adversarial robustness and security / Black-box and transfer attacks
Mutual-Modality Adversarial Attack with Semantic Perturbation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a mutual-modality adversarial attack that uses CLIP as surrogate: a generator attacks the image embedding while prompt tuning defends the text side, iterated to improve black-box transferability.
Problem
Black-box adversarial attacks have limited transferability to unseen architectures and datasets, which depends heavily on surrogate model choice.
Approach
Treats CLIP's aligned image-text space as the surrogate: a generator perturbs images to create semantic mismatch with text embeddings, while prompt tuning (GPT-2 candidate words, saliency-guided) re-matches the text side; the two are iteratively optimized in a GAN-like min-max scheme.
Key results
- On cross-architecture transfer, Ours-Clip reaches overall accuracy 23.3 on ImageNet (vs 31.0 for UAN-Res, 32.8 for BIA-VGG) and 30.5 on CIFAR-10 (vs 41.4 for UAN-Res).
- Integrating the method into GAP, CDA and LTAP raises cross-dataset attack success rates, e.g. GAP on Cosmics to CLIP improves from 20.3 to 30.8.
- Integrated into GAMA, cross-task overall accuracy on ImageNet drops from 21.2 to 19.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the feature loss ℓ_feat (ablation) | CLIP semantic visual attack · CIFAR-10 | overall accuracy: 32.7 -> 24.6 | Helped |
| add the feature loss ℓ_feat (ablation) | CLIP semantic visual attack · ImageNet | overall accuracy: 32.5 -> 16.3 | Helped |
| add the triplet loss ℓ_tri (ablation) | CLIP semantic visual attack · CIFAR-10 | overall accuracy: 31.5 -> 24.6 | Helped |
| add the triplet loss ℓ_tri (ablation) | CLIP semantic visual attack · ImageNet | overall accuracy: 32.0 -> 16.3 | Helped |
| add the classification loss ℓ_cls (ablation) | CLIP semantic visual attack · CIFAR-10 | overall accuracy: 31.5 -> 24.6 | Helped |
| add the classification loss ℓ_cls (ablation) | CLIP semantic visual attack · ImageNet | overall accuracy: 34.9 -> 16.3 | Helped |
| add iterative attack-defense training (ablation) | CLIP semantic visual attack · CIFAR-10 | overall accuracy: 31.3 -> 24.6 | Helped |
| add iterative attack-defense training (ablation) | CLIP semantic visual attack · ImageNet | overall accuracy: 31.4 -> 16.3 | Helped |
| replace random GPT-2 prompts with learned prompt updating (ablation) | CLIP mutual-modality attack · CIFAR-10 | overall accuracy: 31.8 -> 24.6 | Helped |
| replace random GPT-2 prompts with learned prompt updating (ablation) | CLIP mutual-modality attack · ImageNet | overall accuracy: 24.8 -> 16.3 | Helped |
| integrate the mutual-modality method into GAMA | GAMA · ImageNet classification (Pascal-VOC source) | overall accuracy: 21.2 -> 19.9 | Helped |
Surprising findings
- Visual-only attack on CLIP (w/o iteration) fools CLIP strongly but transfers poorly to downstream networks, showing the textual defense is needed for transferability.
Limitations
- Gains on some targets are modest or mixed; e.g., on the cross-dataset SimpleViT/CLIP cases and some ResNet targets the improvement is small.
- BIA-VGG has no CIFAR-10 results, and the ensemble-defense experiment is only in the supplementary.
- Method relies on a single CLIP surrogate and specific prompt/word-saliency choices.
Experimental setup
Datasets: CIFAR-10 ImageNet Comics Paintings ChestX Pascal-VOC
Benchmarks: CIFAR-10 ImageNet
Metrics: classification accuracy (lower is better) overall group-averaged accuracy attack success rate
Baselines: UAN BIA GAMA GAP CDA LTAP
Builds on: CLIP CoOp UAN GAMA Probability Weighted Word Saliency
Closest papers by meaning
- Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks (CVPR 2025 · similarity 0.88)
- A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1 (NeurIPS 2025 · similarity 0.87)
- Pixel2Feature Attack (P2FA): Rethinking the Perturbed Space to Enhance Adversarial Transferability (ICML 2025 · similarity 0.87)
- PubDef: Defending Against Transfer Attacks From Public Models (ICLR 2024 · similarity 0.86)
- Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (ICLR 2024 · similarity 0.86)
- X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.