Research map / Vision-language and multimodal / Multimodal LLMs

Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MM-SafetyBench++, a paired safe/unsafe contextual safety benchmark built from MM-SafetyBench with Qwen-Image-generated images, and EchoSafe, a training-free self-reflective memory framework that retrieves past safety insights to improve MLLM contextual safety.

Problem

MLLMs are evaluated for safety mainly by refusal of explicitly unsafe inputs, which ignores over-defensiveness on benign queries and subtle contextual differences in user intent; existing benchmarks are low-fidelity, unpaired, and rely on binary ASR-style metrics.

Approach

Pair each unsafe image-text query with a minimally edited safe counterpart to flip intent, score both refusal quality and answer quality with a judge, and add a test-time memory where the model self-reflects to store safety insights and retrieves top-k by CLIP embedding similarity to inject into prompts.

Key results

• On MM-SafetyBench++ (Gen mode) with Qwen-2.5-VL, EchoSafe reaches an average CCR of 87.9%, outperforming AdaShield by 16.8%.
• With Qwen-2.5-VL, EchoSafe reduces ASR on MM-SafetyBench from 22.72% (SD) and 25.05% (TYPO) to 0.04% and 0.02%.
• EchoSafe adds about 1.33x inference time and 1.69x FLOPs relative to base while improving performance 2.60x as reported; continual memory yields up to +5.2% CCR gain.

Surprising findings

• Naive safety LoRA fine-tuning (Post-hoc and Mixed LoRA) drives refusal near 100% but answer rates on safe queries to near zero, collapsing the harmonic mean.
• AdaShield attains the highest refusal among prior defenses but substantially degrades answer rate on safe queries, showing over-defense.

Limitations

• Evaluation relies on GPT-5-mini as judge, with only moderate Spearman agreement (0.72-0.74) to Gemini and humans.
• Safe rewrites are generated with GPT-5 and images with Qwen-Image, so synthetic artifacts may bias results.
• Memory is initialized empty per safety category and effectiveness depends on the ordering and similarity of past samples.

Experimental setup

Datasets: MM-SafetyBench MSSBench SIUO MME MMBench ScienceQA TextVQA

Benchmarks: MM-SafetyBench++ MM-SafetyBench MSSBench SIUO MME MMBench ScienceQA TextVQA

Metrics: refusal rate answer rate quality score contextual correctness rate (harmonic mean) attack success rate harmonic mean Safe-and-Effective score

Baselines: FigStep ECSO AdaShield Post-hoc LoRA Mixed LoRA GPT-5 GPT-5-Mini Gemini-2.5-Flash Qwen3-VL-8B

Builds on: MM-SafetyBench FigStep ECSO AdaShield CLIP

Closest papers by meaning

• Multimodal Situational Safety (ICLR 2025 · similarity 0.92)
• When Safe Unimodal Inputs Collide: Optimizing Reasoning Chains for Cross-Modal Safety in Multimodal Large Language Models (AAAI 2026 · similarity 0.90)
• ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.90)
• SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization (CVPR 2026 · similarity 0.90)
• Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models (ICLR 2026 · similarity 0.90)
• GuardAlign: Robust Safety Alignment in Multimodal Large Language Models (ICLR 2026 · similarity 0.89)