Research map / Vision-language and multimodal / Multimodal LLMs

GuardAlign: Robust Safety Alignment in Multimodal Large Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GuardAlign, a training-free input-side defense for MLLMs that uses optimal-transport patch-prompt matching to mask unsafe image regions and attention calibration to keep safety-prefix attention strong during decoding.

Problem

MLLMs still produce unsafe responses to malicious images; CLIP-based input-side detection is inaccurate on complex scenes and safety-prefix signals decay during decoding.

Approach

Replace global CLIP similarity with entropy-weighted optimal transport between image patches and expanded unsafe-prompt variants to locate and mask unsafe patches, and amplify attention from instruction tokens to safety-prefix tokens in middle layers.

Key results

• On SPA-VL, GuardAlign reduces LLaVA-1.5-7B USR from 46.04 (vanilla) and 16.98 (ETA) to 10.31.
• On VQAv2, LLaVA-1.5-7B improves from 78.51 to 79.21 while preserving utility.
• Ablation on LLaVA-1.5-7B SPA-VL: OT detection plus attention calibration gives USR 10.31 vs 32.12 with calibration alone and 46.04 with neither.

Surprising findings

• Conclusion claims USR reductions of up to 90%, whereas the reported tables show reductions of at most about 39% on SPA-VL relative to ETA-level baselines, an internal inconsistency.

Limitations

• Not explored on reasoning-oriented multimodal models.
• Attention calibration requires internal attention access, so black-box use is limited to OT detection.
• Evaluation relies on string-matching ASR and LLM-judge USR, which can have false positives or negatives.

Experimental setup

Datasets: SPA-VL MM-SafetyBench FigStep AdvBench VQAv2 TextVQA MME MMBench ScienceQA-IMG

Benchmarks: SPA-VL MM-SafetyBench FigStep Adv. Image+Text AdvBench VQAv2 TextVQA MME MMBench ScienceQA-IMG SPA-VL Help

Metrics: Unsafe Response Rate (USR) Attack Success Rate (ASR) Helpful Score Win-Tie rate accuracy

Baselines: ECSO ETA Posthoc-LoRA Mixed-LoRA VLGuard

Builds on: ECSO CLIP Sinkhorn algorithm safety-aware prefix

Closest papers by meaning

• Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? (CVPR 2025 · similarity 0.92)
• Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility (ICML 2026 · similarity 0.91)
• ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.91)
• DAVSP: Safety Alignment for Large Vision-Language Models via Deep Aligned Visual Safety Prompt (AAAI 2026 · similarity 0.91)
• Cross-Modal Safety Mechanism Transfer in Large Vision-Language Models (ICLR 2025 · similarity 0.90)
• VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety (ICLR 2026 · similarity 0.90)