Research map / Adversarial robustness and security / Attacks on text and graphs

FusionAudit: Pathway-Conditioned Robustness Auditing for Native Multimodal Models

NeurIPS 2026 · Adversarial robustness and security › Attacks on text and graphs · approach: Vision-language model robustness

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PaRMA/FusionAudit, a diagnostic-driven, pathway-conditioned robustness audit for native VLMs over pixel-perturbation and rendered-text pathways, introducing MANTIS and Nat-Evid attacks, evaluated on 18 open-weight VLMs.

Problem

Existing VLM robustness evaluations use single pixel-centric attacks or typographic overlays in isolation and do not account for how native multimodal fusion architectures change the visual attack surface.

Approach

Defines pathway-conditioned residual risk over pixel and rendered-text pathways, uses diagnostics (Read-Use Gap, Overlay Attention Ratio) to expose architecture-specific blind spots, and uses those diagnostics to design stronger attacks (MANTIS first-divergence margin attack, Nat-Evid host-congruent text insertion) whose union gives a lower bound on risk.

Key results

• Level-mean pathway-complete risk rises from bR0 to bR+ by +10.6% (L1), +16.9% (L2), and +25.1% (L3) across 18 VLMs.
• End-PGD ranking vs. bR+ ranking gives Spearman 0.837 and Kendall tau 0.634, with 8 of 18 models shifting at least three ranks.
• Layer restoration recovery of End-PGD flips falls from 92.2% (L1) to 62.6% (L2) to 25.6% (L3).

Surprising findings

• Native VLMs attend heavily to typographic overlays (OAR more than doubles over clean in L2/L3) yet often do not adopt the injected wrong answer, the see-but-not-deceived signature.
• Absolute residual risk (bR+) decreases in newer, deeper-fusion models even as the diagnostic-driven risk gap persists.

Limitations

• The nativeness taxonomy (L1-L3) is observational and confounded with model family, scale, and training data; not a causal ladder.
• Rendered-text pathway covers only task-relevant wrong-answer injection, not command-following prompt injection or jailbreaks.
• MANTIS requires white-box gradient access; the L4 level is not empirically audited.

Experimental setup

Datasets: ChartQA DocVQA MME

Benchmarks: ChartQA DocVQA MME

Metrics: attack success rate (conditioned on clean-correct set) pathway-complete risk Spearman correlation Kendall tau Read-Use Gap Overlay Attention Ratio

Baselines: Vis-PGD End-PGD Typo-Evid AutoAttack

Builds on: PGD Typo-Evid (Cheng et al.) AutoAttack

Closest papers by meaning

• ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.90)
• Internal Activation Revision: Safeguarding Vision Language Models Without Parameter Update (AAAI 2025 · similarity 0.88)
• VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Models (ICML 2026 · similarity 0.88)
• Jailbreaking Vision-Language Models via Dissonance-Guided Suffix Optimization and Image-Phrase Injection (CVPR 2026 · similarity 0.88)
• FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (AAAI 2025 · similarity 0.87)
• GuardAlign: Robust Safety Alignment in Multimodal Large Language Models (ICLR 2026 · similarity 0.87)