Research map / Adversarial robustness and security / Attacks on text and graphs

HQA-VLAttack: Towards High Quality Adversarial Attack on Vision-Language Pre-Trained Models

NeurIPS 2025 · Adversarial robustness and security › Attacks on text and graphs · approach: Vision-language model robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HQA-VLAttack, a transfer-based black-box attack on vision-language pretrained models that combines counter-fitted-vector text substitutions, layer-importance image initialization, and contrastive image optimization that pushes apart positive pairs and pulls together negative pairs.

Problem

Black-box adversarial attacks on vision-language pre-trained models for image-text retrieval, where existing transfer attacks only reduce positive-pair similarity and ignore negative pairs, and query-based attacks are costly.

Approach

Use counter-fitted word vectors (with BERT-Attack fallback) for semantically consistent word substitutions, weight layer-wise feature disruption by cosine similarity of [CLS] tokens across layers, and apply a contrastive loss on the surrogate model that both decreases positive-pair and increases negative-pair similarity so adversarial images retrieve negatives.

Key results

• On Flickr30K with ALBEF surrogate, HQA-VLAttack reaches 62.88% TR R@1 ASR on TCL vs 51.09% for DRA in black-box transfer, and 100% ASR attacking CLIP-ViT in white-box setting.
• On MSCOCO with CLIP-ViT surrogate attacking CLIP-CNN, ASR gains over DRA are large, e.g. 88.93% TR R@1 vs 64.41% for DRA when attacking ... (ALBEF surrogate attacking CLIP-ViT: 56.05% vs 35.96% TR R@1 for DRA).
• On cross-task transfer from ALBEF, HQA-VLAttack lowers RefCOCO+ Val accuracy to 46.48 from baseline 58.46, outperforming Co-Attack, SGA and DRA.

Measured effects reported in the paper

Limitations

• Text perturbation limited to single-word replacement, restricting attack expressiveness
• Only evaluated on retrieval/grounding/captioning with a few VLP models and datasets
• No error bars or statistical significance reported

Experimental setup

Datasets: Flickr30K MSCOCO RefCOCO+

Benchmarks: Flickr30K MSCOCO RefCOCO+

Metrics: attack success rate TR R@1 IR R@1 BLEU-4 METEOR ROUGE-L CIDEr SPICE

Baselines: PGD BERT-Attack Sep-Attack Co-Attack SGA DRA

Builds on: SGA DRA BERT-Attack PGD ALBEF

Closest papers by meaning

• Attention! Your Vision Language Model Could Be Maliciously Manipulated (NeurIPS 2025 · similarity 0.89)
• Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks (CVPR 2025 · similarity 0.88)
• VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models (AAAI 2024 · similarity 0.88)
• Ask, Attend, Attack: An Effective Decision-Based Black-Box Targeted Attack for Image-to-Text Models (NeurIPS 2024 · similarity 0.87)
• Anyattack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models (CVPR 2025 · similarity 0.86)
• A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1 (NeurIPS 2025 · similarity 0.86)