Research map / Vision-language and multimodal / Visual question answering

VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models

AAAI 2024 · Vision-language and multimodal › Visual question answering · approach: Attention and prior-robust VQA · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VQAttack, a transferable adversarial image-text attack on VQA models that uses only a pre-trained multimodal source model, combining LLM-enhanced masked answer anti-recovery image perturbations with iterative cross-modal joint image and synonym-based text perturbations.

Problem

Adversarial robustness of the "pre-training & fine-tuning" VQA paradigm is unexplored; attacks must transfer from a pre-trained source VL model to unseen fine-tuned black-box VQA target models while jointly perturbing continuous images and discrete text.

Approach

Generate image perturbations via latent-feature disruption plus an LLM-built masked answer anti-recovery loss using the source model's MLM head, and iteratively update image and text perturbations jointly (text via BERT candidate synonyms ranked by embedding-gradient similarity), with a heuristic trigger controlling when text is attacked.

Key results

• VQAttack achieves average ASR of 22.49% with ViLT as source, 34.23% with TCL, 31.88% with ALBEF, 29.33% with VLMO-B and 25.51% with VLMO-L across transfer settings.
• In the shared-structure setting (Table 3), VQAttack reaches up to 79.00% ASR on VQAv2 (ViLT) and 66.20% on TextVQA (VLMO-B), outperforming Co-Attack on all listed targets.
• Ablation on TCL-to-VLMO-B shows the multimodal-encoder loss gives large ASR gains, and LLM-enhanced loss and the joint attack module each add further improvement.

Measured effects reported in the paper

Surprising findings

• Larger source model VLMO-L yields slightly lower ASR than VLMO-B as source, which the authors interpret as larger models being more adversarially robust.
• Under the shared-structure setting, the text-only baselines and image-only baselines are relatively weak compared to the joint attack, and LLM-E sometimes underperforms LRP on ViLT for TextVQA.

Limitations

• Evaluation limited to five VL models and two VQA datasets with 6,000 and 1,000 sampled correctly predicted examples.
• Text-attack baselines were adapted by querying the source-fine-tuned VQA model, so comparison is not fully like-for-like.
• Relies on the source model having an MLM head and on ChatGPT-4 to build masked sentences, limiting applicability to models without these components.

Experimental setup

Datasets: VQAv2 TextVQA

Benchmarks: VQAv2 TextVQA

Metrics: attack success rate

Baselines: BERT-Attack Rewrite-Rollback DR SSP FDA Co-Attack

Builds on: Co-Attack BERT-Attack PGD DIM BadEncoder

Closest papers by meaning

• Human-Adversarial Visual Question Answering (NeurIPS 2021 · similarity 0.92)
• A Novel Framework for Robustness Analysis of Visual QA Models (AAAI 2019 · similarity 0.92)
• FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.90)
• On the Robustness of Large Multimodal Models Against Image Adversarial Attacks (CVPR 2024 · similarity 0.89)
• When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.89)
• NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.89)