Research map / Vision-language and multimodal / Multimodal LLMs

Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the MIS multi-image safety dataset with safety chain-of-thought labels and the MIRage fine-tuning method, which improves multi-image safety and general capabilities of InternVL2.5-8B.

Problem

Existing safety fine-tuning methods for VLMs fail on challenging situational safety cases where harmful intent arises from neutral image-text combinations, and often cause over-refusal that harms helpfulness; multi-image safety is largely unevaluated.

Approach

Fine-tune VLMs on multi-image safety instruction data whose responses follow a safety chain-of-thought (perceive both images, reason about harmful intent, then answer safely), with only a small amount of general QA data, so safety visual reasoning improves without over-refusal.

Key results

• InternVL2.5-8B+MIRage reduces ASR on MIS-easy/hard/real to 0.24%/0.20%/0.00% from roughly 80%/85%/76% for the base model.
• MIRage raises average general benchmark accuracy from 60.47 to 61.30 across five general tasks, with no trade-off on helpfulness.
• On SIUO, MIRage reaches 71.26 safe rate versus 24.85 for the base InternVL2.5-8B; VLGuard-R reaches 64.23.

Surprising findings

• Textual SFT and VLGuard-P push models toward over-refusal, with VLGuard-P rejecting even white-image inputs at nearly 50% rate.
• Fine-tuning on MIS with only 500 general samples (11% of unsafe data) slightly outperforms the base model on general benchmarks.

Limitations

• Model remains vulnerable to certain adversarial attacks (stated in ethics statement).
• Safety CoT labels are a simple construction; more complex reasoning labels and RL are left to future work.
• Safety evaluation relies on GPT-4o and LlamaGuard3-8B as judges.

Experimental setup

Datasets: MIS training split SafeRLHF VLGuard M4-Instruct LAION-2B-en-research-safe FigStep MSSBench MM-SafetyBench JailBreakV-28K SIUO MOSSBench

Benchmarks: MIS test set (easy, hard, real) MMStar MMMU MMT-Bench MuirBench Q-Bench FigStep MSSBench MM-SafetyBench JailBreakV-28K SIUO MOSSBench

Metrics: ASR RSR RR HR exact match accuracy safe rate accuracy on MSSBench safe/unsafe

Baselines: Textual SFT VLGuard-P VLGuard-R VLGuard-M GPT-4o Gemini-1.5-pro Qwen2-VL-72B-Instruct LLaVA-OV-72B-Chat InternVL2.5-78B

Builds on: VLGuard Textual SFT SafeRLHF M4-Instruct

Closest papers by meaning

• VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety (ICLR 2026 · similarity 0.93)
• Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models (ICML 2024 · similarity 0.91)
• ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.90)
• Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection (CVPR 2026 · similarity 0.90)
• How Does Vision-Language Adaptation Impact the Safety of Vision Language Models? (ICLR 2025 · similarity 0.90)
• Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory (CVPR 2026 · similarity 0.90)