Research map / Adversarial robustness and security / Attacks on text and graphs

Anyattack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models

CVPR 2025 · Adversarial robustness and security › Attacks on text and graphs · approach: Vision-language model robustness · 16 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AnyAttack, a self-supervised pre-trained-then-fine-tuned noise generator (decoder on frozen CLIP ViT-B/32 embeddings) that crafts transferable targeted adversarial images against VLMs without target labels.

Problem

Targeted adversarial attacks on vision-language models require specific target labels or target images for supervision, which limits scalability and real-world impact.

Approach

Self-supervised pre-training of a decoder that maps CLIP image embeddings to adversarial noise, so that noise added to random unrelated images is embedded like the original image (using the image itself as supervision); then fine-tuned on downstream datasets with cosine or bidirectional contrastive loss and auxiliary ensemble models.

Key results

• With auxiliary models, AnyAttack-Bi w/ Aux reaches R@Mean 35.88 on ViT-B/16 retrieval on MSCOCO, versus 20.86 for the best baseline (SU-Cos).
• On SNLI-VE classification, AnyAttack-Cos w/ Aux reaches 44.8% accuracy versus 24.8% for the strongest baseline (SASD-WS-MSE).
• On commercial VLMs, AnyAttack-Cos w/ Aux achieves ASR 31% on Google Gemini and 38% on OpenAI GPT (100 MSCOCO targets), versus at most 5% and 28% for baselines.

Measured effects reported in the paper

Surprising findings

• Transfer-based attacks crafted on a CLIP ViT-B/32 surrogate produce targeted responses in commercial VLMs such as Gemini and GPT-4o mini without any target labels.

Limitations

• Attacks are transfer-based with a single surrogate family (CLIP ViT-B/32) plus auxiliary models, and effectiveness on commercial systems is measured on small samples (100 targets) with a relevance-judgment prompt.
• Perturbation budget fixed at epsilon 16/255 under L-infinity norm; no defense evaluation reported.

Experimental setup

Datasets: LAION-400M MSCOCO Flickr30K SNLI-VE ImageNet-1K

Benchmarks: MSCOCO Flickr30K SNLI-VE

Metrics: attack success rate recall@K (TR@1, IR@1, R@Mean) accuracy SPICE BLEU-4 METEOR ROUGE-L CIDEr

Baselines: AttackVLM-ii AttackVLM-it SU-Cos SU-MSE SASD-WS-Cos SASD-WS-MSE

Builds on: AttackVLM SU SASD-WS CLIP

Closest papers by meaning

• Pandora's Box: Towards Building Universal Attackers against Real-World Large Vision-Language Models (NeurIPS 2024 · similarity 0.88)
• Attention! Your Vision Language Model Could Be Maliciously Manipulated (NeurIPS 2025 · similarity 0.88)
• Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks (CVPR 2025 · similarity 0.88)
• Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models (CVPR 2025 · similarity 0.87)
• A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1 (NeurIPS 2025 · similarity 0.87)
• HQA-VLAttack: Towards High Quality Adversarial Attack on Vision-Language Pre-Trained Models (NeurIPS 2025 · similarity 0.86)