Research map / Vision-language and multimodal / Multimodal LLMs

DAVSP: Safety Alignment for Large Vision-Language Models via Deep Aligned Visual Safety Prompt

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DAVSP, which pads images with a trainable visual safety prompt and trains it via a harmfulness-vector activation-space margin loss plus output CE, improving LVLM resistance to malicious queries while preserving benign utility.

Problem

LVLMs remain vulnerable to malicious multimodal queries; existing lightweight defenses (textual safety prompts, additive visual perturbations) resist attacks insufficiently and degrade benign utility, and rely on shallow response-level supervision.

Approach

Replace additive pixel perturbations with a trainable padding region around a resized image (visual safety prompt) and supervise it in the activation space by projecting hidden states onto a harmfulness vector (mean malicious-minus-benign activation difference at a middle layer) with margin-based loss.

Key results

• On LLaVA-1.5-13B, DAVSP reaches 98.72% RSR on MM-SafetyBench SD+TYPO and 84.20% on FigStep, versus 91.96% and 70.80% for the best prior baseline ESIII.
• On Qwen2-VL-7B-Instruct, DAVSP reaches 99.20% RSR on FigStep, and MM-Vet total 61.61 versus 61.64 for AdaShield-A and 55.93 for ESIII.
• Transferring the prompt trained on LLaVA-1.5-13B to LLaVA-1.5-7B raises FigStep RSR to 100.00% (vs 99.40% for text-prompt only).

Surprising findings

• Replacing the visual safety prompt with additive perturbations reduces MME-P from 1318 to 1228 on LLaVA-1.5-13B, showing additive perturbations harm perception.
• Keyword-based refusal evaluation is found to produce high false positives due to shallow alignment, motivating LLM-based judging.

Limitations

• Evaluated only on 7B and 13B models due to compute constraints; scalability to larger models untested.
• Adaptive attack evaluation shows RSR drops to 71% with DAVSP under an adversary targeting its mechanism.

Experimental setup

Datasets: VLGuard MM-SafetyBench MM-Vet FigStep LLaVA-Bench (In-the-Wild) MME

Benchmarks: MM-SafetyBench FigStep MM-Vet MME LLaVA-Bench (In-the-Wild)

Metrics: Resist Success Rate (RSR) utility score

Baselines: AdaShield-S AdaShield-A PAT UniGuard ESIII ECSO

Builds on: AdaShield ESIII UniGuard PAT visual prompt tuning

Closest papers by meaning

• ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.92)
• Cross-Modal Safety Mechanism Transfer in Large Vision-Language Models (ICLR 2025 · similarity 0.91)
• GuardAlign: Robust Safety Alignment in Multimodal Large Language Models (ICLR 2026 · similarity 0.91)
• Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models (ICML 2024 · similarity 0.91)
• SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Models (CVPR 2025 · similarity 0.90)
• Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? (CVPR 2025 · similarity 0.90)