Research map / Adversarial robustness and security / Black-box and transfer attacks

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP

ICML 2025 · Adversarial robustness and security › Black-box and transfer attacks · approach: Transferable adversarial examples · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes X-Transfer, a black-box universal adversarial perturbation attack on CLIP that scales surrogate ensembles efficiently by UCB-selecting top-k encoders per step from a large search space, achieving "super transferability" across data, domains, models, and tasks.

Problem

CLIP encoders and the VLMs built on them are vulnerable to adversarial perturbations, but existing universal attacks do not transfer simultaneously across data, domains, models, and tasks; ensembling many fixed surrogates is computationally expensive.

Approach

Treat each candidate CLIP surrogate as a bandit arm and use UCB over a moment-averaged loss reward to select only k of N surrogates per optimisation step, so the UAP is updated with a small ensemble while the search space scales to 16-64 encoders; the objective is a generic embedding-level loss agnostic to architecture and training objective.

Key results

• Large (N=64) X-Transfer achieves 75.6% average non-targeted ASR on zero-shot classification across 9 black-box CLIP encoders, versus 50.8% for the best baseline (Meta-UAP ensemble).
• On image-text retrieval, Large X-Transfer reaches 71.8 TR@1 and 65.8 IR@1 ASR, compared with 43.8 and 35.7 for the best baseline.
• On VLM captioning, X-Transfer Large reaches 53.3% on OpenFlamingo-3B MSCOCO and 64.1% on BLIP-2 MSCOCO, versus best baseline values of 22.9 and 19.9 for OF-3B and 13.0 for BLIP-2 respectively.

Measured effects reported in the paper

Surprising findings

• A vanilla single-surrogate objective without ensembling performs on par with specialised CLIP-specific UAP baselines.
• Non-targeted UAPs from CLIP encoders show building-like semantic patterns, unlike typical non-targeted perturbations.

Limitations

• The attack relies on pre-trained CLIP encoders as surrogates, and the paper's analysis suggests transfer to convolution-based victims from RN surrogates is weak.
• Victim VLMs are evaluated with a fixed epsilon of 12/255 under L_inf, and robustness of the attack to defenses is not examined in the text provided.

Experimental setup

Datasets: ImageNet CIFAR-10 CIFAR-100 Food GTSRB Stanford Cars STL10 SUN397 MSCOCO Flickr-30K OK-VQA VizWiz

Benchmarks: CIFAR-10 CIFAR-100 Food GTSRB ImageNet Stanford Cars STL10 SUN397 MSCOCO retrieval MSCOCO captioning Flickr-30K OK-VQA

Metrics: non-targeted attack success rate CIDEr VQA accuracy

Baselines: GD-UAP TRM-UAP Meta-UAP AdvCLIP C-PGC ETU

Builds on: universal adversarial perturbations ensemble transfer attacks upper confidence bound bandits

Closest papers by meaning

• LRS: Enhancing Adversarial Transferability through Lipschitz Regularized Surrogate (AAAI 2024 · similarity 0.88)
• Enhancing Adversarial Transferability with Adversarial Weight Tuning (AAAI 2025 · similarity 0.88)
• Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferability (NeurIPS 2025 · similarity 0.88)
• Enhancing Adversarial Transferability with Checkpoints of a Single Model's Training (CVPR 2025 · similarity 0.87)
• CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP (CVPR 2025 · similarity 0.87)
• AIM: Additional Image Guided Generation of Transferable Adversarial Attacks (AAAI 2025 · similarity 0.87)