Research map / Vision-language and multimodal / Multimodal LLMs

How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that vision-language adaptation of LLaMA-2 Chat and Tulu-2 degrades safety even with filtered data, that safety SFT and DPO trade off against helpfulness, and that merging a safety-tuned model with a multimodal-strong model recovers a better balance.

Problem

VL adaptation that turns LLMs into LVLMs often erodes the safety alignment of the original LLM, and existing safety tuning (safety SFT, RLHF) either fails to fully restore safety or over-refuses benign queries, hurting helpfulness.

Approach

Attributes safety degradation to the VL adaptation process itself (not unsafe data), links it to changes in LLM safety layers, and proposes linearly merging a safety-SFT model with a multimodally strong model as a cheap way to balance safety and helpfulness.

Key results

• LLaMA-2-Chat-VL average text-only ASR rises from 10.4% (LLaMA-2 Chat 7B) to 54.4% and multimodal ASR reaches 96.2%, despite filtered safe training data.
• Sequential safety SFT (LLaMA-2-Chat-VL-SL) drives ASR to 0.7% on text-only and 0.7% on multimodal benchmarks but raises XSTest refusal rate to 73.2% and drops multimodal accuracy to 42.3 average.
• Linear merging of LLaMA-2-Chat-VL-SL and LLaVA-Chatty at alpha=0.4 gives a balanced result (average score 80.5 vs. 81.1 for the Tulu-2 variant), keeping safety near SL while retaining multimodal accuracy near the Chatty model.

Measured effects reported in the paper

Surprising findings

• Safety degradation occurs during VL adaptation even when the training data has been filtered for unsafe text and images.
• Sequential safety SFT, which best reduces ASR, causes the largest over-refusal and multimodal accuracy loss.

Limitations

• Experiments are mostly on LLaMA-2-based 7B models, so generalization to other architectures is untested.
• Merging depends on compatibility of the merged models and may underperform with differing training regimes or safety alignments.
• Only harmfulness and exaggerated refusal are studied; cultural and ethical biases are left unexplored.

Experimental setup

Datasets: LLaVA-Pretrain LLaVA-Instruct VLGuard SafeRLHF SPA-VL WildVision-Chat-46k

Benchmarks: SorryBench WildJailbreak MM-SafetyBench SIUO FigStep XSTest MMBench MME SEEDBench TriviaQA BBH HellaSwag

Metrics: attack success rate refusal rate exact match accuracy

Baselines: LLaMA-2-Chat 7B LLaMA-2-Chat-VL-MTL LLaMA-2-Chat-VL-SL Tulu-2-VL-SafeRLHF Tulu-2-VL-SPA-VL LLaMA-2-Chat-VL-SPPFT TIES merging DARE merging

Builds on: LLaVA v1.5 VLGuard SPA-VL Safely Partial-Parameter Fine-Tuning Model Soup TIES-Merging DARE DPO

Closest papers by meaning

• Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models (ICML 2024 · similarity 0.92)
• Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models (ICLR 2026 · similarity 0.90)
• SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Models (CVPR 2025 · similarity 0.90)
• Understanding and Rectifying Safety Perception Distortion in VLMs (NeurIPS 2025 · similarity 0.89)
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.89)
• ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.88)