Research map / Vision-language and multimodal / Multimodal LLMs

SDEval: Safety Dynamic Evaluation for Multimodal Large Language Models

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SDEval, a framework that generates dynamic variants of existing MLLM safety benchmarks (text, image, and text-image perturbations) to reduce data contamination and raise complexity, showing safety scores drop across many MLLMs.

Problem

Static MLLM safety benchmarks suffer from data leakage into training sets, fixed complexity, and become outdated as attacks evolve, so reliable safety evaluation is hard.

Approach

Generate new text-image samples from existing safety benchmarks via controllable text dynamics (word replacement, paraphrasing, typos, linguistic mixing, CoT), image dynamics (augmentation, diffusion-based generation, GPT-4o-guided editing), and cross-modal dynamics (text-to-image, image-to-text, FigStep, HADES), with a validator agent checking semantic consistency.

Key results

• On Dynamic MLLMGuard, InternVL-3-78B's safety (PAR) fell by 17.64 points versus vanilla (39.04 to 21.40 on its average), with GPT-4o PAR dropping 15.67 points.
• FigStep was the most damaging dynamic strategy on MLLMGuard (InternVL-Chat-V1.5 PAR 17.08 vs 40.19 vanilla), while Word Replacement was the most damaging text strategy.
• On Dynamic VLSBench, all 18 evaluated models showed lower safety rates, e.g., GPT-4o safety from 58.50 to 52.83 and Gemini-2.5-Pro from 38.15 to 29.94.

Surprising findings

• Larger parameter count does not reliably improve safety under dynamic evaluation; some InternVL scaling leads to lower safety.
• Capability benchmarks (MMVet, MMBench) also drop under dynamic perturbation, but safety drops are larger for most models.

Limitations

• Relies on GPT-4o for generation and validation, which may introduce bias or errors in semantic consistency checks.
• Contamination analysis uses CLIP similarity with a 0.95 threshold, a coarse proxy for training-data overlap.
• Evaluation covers only a subset of models and relies on a single judge for harmfulness scoring.

Experimental setup

Datasets: MLLMGuard VLSBench MMBench MMVet

Benchmarks: MLLMGuard VLSBench MMBench MMVet

Metrics: ASD PAR Safety Rate accuracy

Baselines: VLB

Builds on: DyVal DME FigStep HADES MLLMGuard

Closest papers by meaning

• MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.93)
• Multimodal Situational Safety (ICLR 2025 · similarity 0.90)
• Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? (CVPR 2025 · similarity 0.89)
• SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.89)
• When Understanding Becomes a Risk: Authenticity and Safety Risks in the Emerging Image Generation Paradigm (CVPR 2026 · similarity 0.89)
• Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory (CVPR 2026 · similarity 0.88)