Research map / Vision-language and multimodal / Multimodal LLMs
Multimodal Situational Safety
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MSSBench, a 1,960-pair benchmark testing whether MLLMs judge query safety from visual context in chat and embodied settings, and shows multi-agent pipelines improve safety.
Problem
MLLMs as multimodal assistants often answer language queries that are unsafe given the visual situation, while the query itself is benign; existing MLLM safety benchmarks focus on clearly unsafe text or adversarial images.
Approach
Defines multimodal situational safety, where the safety of a benign-looking query depends on the image, and builds balanced safe/unsafe image-query pairs for chat and embodied scenarios, plus diagnostic settings (query/intent classification, self-caption, ground-truth caption) and a multi-agent pipeline that splits intent reasoning, visual captioning, safety judgment and answering.
Key results
- Best model Claude 3.5 Sonnet reaches only 64.0% average accuracy on MSSBench under instruction following, with most MLLMs near 50% due to near-zero unsafe-situation accuracy.
- Explicit safety reasoning raises unsafe-situation accuracy but causes over-sensitivity; GPT-4o reaches 90.0% average in the best diagnostic setting (IC w/ GT Cap for chat/embodied combined).
- Multi-agent pipeline lifts open-source models (e.g., Qwen-VL chat binary avg 77.2% vs 52.8% baseline-style settings) to roughly Gemini-level performance on chat tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add explicit safety reasoning about the query or intent (ablation) | MLLMs on MSSBench · MSSBench | accuracy | Helped |
| Add explicit safety reasoning about the query or intent (ablation) | MLLMs on MSSBench safe situations · MSSBench safe situations | accuracy | Hurt |
| Classify the user's intent rather than the query (ablation) | MLLMs · MSSBench Chat Task | accuracy | Helped |
| Add a model-generated caption to intent classification (ablation) | MLLMs · MSSBench Chat Task | accuracy | Helped |
| Add a model-generated caption to intent classification (ablation) | MLLMs · MSSBench Embodied Task | accuracy | Hurt |
| Provide ground-truth situation information for intent classification (ablation) | MLLMs · MSSBench | accuracy | Helped |
| Split safety subtasks across agents in the multi-agent pipeline | MLLMs · MSSBench | accuracy | Helped |
| Replace the reasoned environment state with the ground-truth environment state (ablation) | Claude and GPT4o multi-agent · MSSBench Embodied Task | accuracy: Claude: 59.0 -> 76.3; GPT4o: 59.6 -> 82.2 | Helped |
| Replace the reasoned environment state with the ground-truth observation (ablation) | Claude and GPT4o multi-agent · MSSBench Embodied Task | accuracy: Claude: 59.0 -> 83.6; GPT4o: 59.6 -> 87.1 | Helped |
Surprising findings
- Most MLLMs comply with nearly all queries in unsafe situations, yielding around 50% average accuracy close to random.
- Explicit safety reasoning improves unsafe detection but lowers safe-situation accuracy, indicating over-sensitivity.
Limitations
- Multi-agent pipelines add latency due to multiple reasoning steps.
- Embodied performance remains far from perfect even with multi-agent and ground-truth state ablations.
- Evaluation relies on GPT-4o to categorize responses into safety levels.
Experimental setup
Datasets: COCO MSSBench
Benchmarks: MSSBench
Metrics: accuracy safe accuracy unsafe accuracy
Baselines: LLaVA-1.6 MiniGPT-4v2 Qwen-VL DeepSeek mPLUG-Owl2 Claude 3.5 Sonnet GPT-4o Gemini Pro-1.5
Builds on: COCO DALL-E 3
Closest papers by meaning
- Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory (CVPR 2026 · similarity 0.92)
- MLLMs Fail to Refuse when Using Tools Agentically (NeurIPS 2026 · similarity 0.91)
- LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Models (CVPR 2026 · similarity 0.91)
- MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.91)
- When Safe Unimodal Inputs Collide: Optimizing Reasoning Chains for Cross-Modal Safety in Multimodal Large Language Models (AAAI 2026 · similarity 0.91)
- VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.