Research map / Adversarial robustness and security / Attacks on text and graphs

ReaLM: A Unified Red-Teaming Benchmark for Physical-World VLMs

NeurIPS 2026 · Adversarial robustness and security › Attacks on text and graphs · approach: Vision-language model robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ReaLM, a unified black-box red-teaming benchmark for physical-world VLMs that integrates 12 attacks, 3 defenses, and 13 VLMs on 832 scene-specific cases with agentic target generation.

Problem

Red-teaming methods for VLMs are evaluated inconsistently across datasets, metrics, and threat models, and existing red-teaming benchmarks focus on content safety rather than physically grounded functional failures in embodied VLMs.

Approach

A unified black-box protocol with shared datasets, metrics, and threat model, plus an agentic VLM pipeline that generates scenario-specific wrong answers and target images so that diverse attack families are compared under aligned adversarial objectives.

Key results

• PromptInject achieves the highest average ASR across models at 64.8%, while CoA is the strongest visual perturbation attack at 54.6%.
• AnyAttack (single forward pass) reaches 49.9% average ASR, slightly above FOA's 48.1% iterative baseline.
• BlueSuffix reduces PromptInject ASR from 65.6% to 43.9% on average across 9 models, while no defense generalizes across all attack paradigms.

Surprising findings

• Model scale alone does not confer robustness: Qwen3.5 scaling from 9B to 122B barely changes PromptInject ASR (63.7% to 64.3%).
• Cosmos-Reason models, post-trained for physical reasoning, show among the highest vulnerability (Cosmos-Reason2-8B reaches 76.8% PromptInject ASR).

Limitations

• Evaluates only single-image, single-turn scenarios; no multi-frame video.
• Defense evaluation covers only model-agnostic preprocessing; adversarial training not evaluated.
• Digital stress tests and injection attacks assume broader intervention surfaces than camera-only physical settings.

Experimental setup

Datasets: physical-AI benchmarks (unspecified)

Benchmarks: ReaLM

Metrics: Attack Success Rate clean accuracy

Baselines: FOA-Attack M-Attack V-Attack CoA PhysPatch AdvDiffVLM AdvEDM-A AnyAttack FigStep PromptInject ImageMix

Builds on: HarmBench M-Attack Chain-of-Attack FigStep BlueSuffix

Closest papers by meaning

• TRUST-VLM: Thorough Red-Teaming for Uncovering Safety Threats in Vision-Language Models (ICML 2025 · similarity 0.92)
• RedVLA: Physical Red Teaming for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.91)
• TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Exploration (CVPR 2026 · similarity 0.90)
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (ICML 2024 · similarity 0.89)
• Automated Red Teaming with GOAT: the Generative Offensive Agent Tester (ICML 2025 · similarity 0.88)
• AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions (CVPR 2026 · similarity 0.88)