Research map / Vision-language and multimodal / Multimodal LLMs

VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VLSU, a 8,187-sample human-annotated real-image benchmark for joint vision-language safety with a borderline severity class and 17 severity combination patterns, showing large joint-understanding and over-block/under-refuse gaps in 11 VLMs.

Problem

Existing multimodal safety evaluations treat image and text separately, miss harm emerging from joint interpretation, and do not distinguish borderline (educational) content from clearly unsafe content, leading to over-blocking or under-refusal.

Approach

A formal safety severity framework (safe/borderline/unsafe) with per-modality and joint labels yielding 17 combinatorial patterns, built with a multi-stage pipeline that retrieves real web images and synthesizes parameterized queries, followed by triple human annotation.

Key results

• Models reach 90%+ accuracy on clear unimodal unsafe patterns but only 20-55% on joint-understanding patterns (S-S-U).
• 34% of joint-classification errors occur even when both individual modalities were classified correctly.
• Helpful instruction framing cuts Gemini-1.5 borderline over-refusal from 62.4% to 10.4% but drops unsafe refusal from 90.8% to 53.9%.

Measured effects reported in the paper

Surprising findings

• Structured chain-of-thought prompting gives no gain for the strongest models (Gemini-1.5, Gemini-2.5, Qwen2.5VL-32B), within 1%.
• Best existing benchmark F1 is near saturated (up to 98.6% on MSTS) while best VLSU F1 is only 70.9%.

Limitations

• English-only text.
• Dataset release is gated access rather than fully open.
• Image modality labels by a single author; synthetic query generation via Gemini-1.5-Pro.

Experimental setup

Datasets: VLSU

Benchmarks: VLSU MM-SafetyBench VLSBench MSTS

Metrics: accuracy F1 refusal rate

Baselines: MM-SafetyBench VLSBench MSTS GPT-4o Gemini-1.5 Gemini-2.5 Qwen2.5VL

Builds on: MM-SafetyBench VLSBench MSTS DSPy LlavaGuard

Closest papers by meaning

• Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models (ICLR 2026 · similarity 0.93)
• ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.92)
• ELITE: Enhanced Language-Image Toxicity Evaluation for Safety (ICML 2025 · similarity 0.91)
• SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Models (CVPR 2025 · similarity 0.91)
• Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection (CVPR 2026 · similarity 0.91)
• Multimodal Situational Safety (ICLR 2025 · similarity 0.91)