Research map / Adversarial robustness and security / Black-box and transfer attacks

A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1

NeurIPS 2025 · Adversarial robustness and security › Black-box and transfer attacks · approach: Transfer attacks on face recognition · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes M-Attack, a transfer-based targeted attack on closed-source LVLMs that random-crops the source image, aligns local crops with target embeddings via a CLIP ensemble, and reports >90% success on GPT-4o and o1.

Problem

Transfer-based targeted adversarial attacks against closed-source commercial vision-language models often fail, because the perturbations look uniform and lack clear semantic content.

Approach

Align randomly cropped and resized local regions of the adversarial source image with the target image (local-to-global or local-to-local) in CLIP embedding space, aggregating local perturbations into a semantically focused global perturbation, with an ensemble of surrogate CLIP models; also introduces KMRScore for keyword-based evaluation.

Key results

• On GPT-4o, the ensemble attack reaches 0.95 ASR and 0.82 KMR_a at epsilon=16, versus 0.42 for AnyAttack.
• Attacking o1 reaches 0.94 ASR (Table on reasoning LVLMs), and GPT-4.5 reaches 0.95 ASR, exceeding 90% as claimed.
• On 1K images at threshold 0.3, ASR on GPT-4o is 0.868 for the method versus 0.419 for AnyAttack.

Measured effects reported in the paper

Surprising findings

• Reasoning models (o1, Claude-3.7-thinking, Gemini-2.0-flash-thinking) show equal or weaker robustness than their non-reasoning counterparts, attributed to shared vision components.
• Global-to-global matching achieves the highest feature similarity yet the lowest transfer ASR, suggesting optimized distance alone does not drive transferability.

Limitations

• Authors acknowledge that newer and more robust commercial models may reduce effectiveness, so the approach may not maintain its success rate.
• Evaluation relies on GPT-4o-based keyword matching and GPT-4o-based similarity scoring, with KMRScore partially manual and manual review only on 20% of outputs.
• Experiments use only 100 images at 224x224 for the main comparison; no error bars are reported.

Experimental setup

Datasets: NIPS 2017 Adversarial Attacks and Defenses Competition dataset

Benchmarks: NIPS 2017 Adversarial Attacks and Defenses Competition dataset

Metrics: KMRScore (KMR_a, KMR_b, KMR_c) ASR (GPTScore > 0.3) normalized l1 norm normalized l2 norm

Baselines: AttackVLM SSA-CWA AnyAttack AdvDiffVLM

Builds on: I-FGSM MI-FGSM PGD AttackVLM SSA-CWA

Closest papers by meaning

• Pandora's Box: Towards Building Universal Attackers against Real-World Large Vision-Language Models (NeurIPS 2024 · similarity 0.91)
• Stage-wise Attention-Guided Region Sequencing for Adversarial Attacks on Large Vision-Language Models (NeurIPS 2026 · similarity 0.90)
• Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks (CVPR 2025 · similarity 0.90)
• Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probing (NeurIPS 2025 · similarity 0.89)
• SLADE: Shielding against Dual Exploits in Large Vision-Language Models (CVPR 2025 · similarity 0.88)
• Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models (CVPR 2025 · similarity 0.88)