Research map / Vision-language and multimodal / Multimodal LLMs
SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds SPA-VL, a ~100K-sample automatically constructed safety preference dataset (question, image, chosen, rejected) for VLM RLHF, and shows DPO/PPO on LLaVA-1.5 improve harmlessness and helpfulness.
Problem
VLMs have weak safety alignment through the visual modality, and there is a lack of large-scale, high-quality open preference datasets for safety alignment of vision-language models.
Approach
A fully automated pipeline builds a multimodal safety preference dataset covering 6 harm domains, 13 categories and 53 subcategories, with easy questions, hard questions and hard statements per image, and responses from 12 VLMs selected for safety-level diversity and annotated by GPT-4V for harmlessness and helpfulness.
Key results
- LLaVA+SPA-VL-DPO and PPO reduce MM-SafetyBench average ASR from 20.54 to about 0.5-0.6 and AdvBench vanilla ASR from 98.08 to 0 (DPO) and 97.88 (PPO), with HarmEval USR dropping from 44.15 to 0.
- Scaling data from 100 to 90K steadily lowers HarmEval harm score and ASR while raising Anthropic-Helpful scores; PPO at 90K reaches over 60% HelpEval win rate against GPT-4V.
- General VQA and multimodal benchmarks (e.g., MMBench, SEED-Bench) remain close to the LLaVA-7b base model after SPA-VL alignment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train LLaVA-7B with SPA-VL using DPO | LLaVA-7B · MM-SafetyBench Avg | ASR: 20.54 -> 0.6 | Helped |
| Train LLaVA-7B with SPA-VL using PPO | LLaVA-7B · AdvBench suffix | ASR: 99.81 -> 2.12 | Helped |
| Increase PPO training data from 100 to 90K (ablation) | LLaVA-1.5 (7B) · Anthropic-Helpful | help score: 51.50 -> 70.00 | Helped |
| Increase DPO training data from 100 to 90K (ablation) | LLaVA-1.5 (7B) · AdvBench suffix | ASR: 99.62 -> 0.00 | Helped |
| Use response pairs from all 12 models instead of only safe responses (ablation) | LLaVA-1.5 (7B) trained with DPO · AdvBench suffix | ASR: 65.38 -> 6.54 | Helped |
| Use response pairs from all 12 models instead of only unsafe responses (ablation) | LLaVA-1.5 (7B) trained with DPO · AdvBench suffix | ASR: 60.77 -> 6.54 | Helped |
| Combine easy questions, hard questions, and hard statements instead of using one type (ablation) | LLaVA-1.5 (7B) trained with DPO · AdvBench suffix | ASR: 24.04 -> 6.54; 11.54 -> 6.54; 5.00 -> 6.54 | Helped |
| Add the projection layer to DPO training (ablation) | LLaVA-1.5 (7B) · MM-SafetyBench Avg | ASR: 1.64 -> 0.60 | Helped |
| Add the projection layer to DPO training (ablation) | LLaVA-1.5 (7B) · AdvBench suffix | ASR: 0.19 -> 0.00 | No clear effect |
| Add the projection layer to PPO training (ablation) | LLaVA-1.5 (7B) · MM-SafetyBench Avg | ASR: 0.89 -> 0.74 | Mixed |
| Align LLaVA-7B with SPA-VL using DPO | LLaVA-7B · POPE | F1 score: 85.85 -> 78.59 | Hurt |
Surprising findings
- Safe-only response pairs trained worse on adversarial suffix attacks than mixed safe/unsafe pairs, showing that response diversity matters.
- Training with the projection layer unfrozen improved image-involved safety tests but DPO reduced POPE F1 from 85.85 to 78.59.
Limitations
- Preference labels come from GPT-4V with Gemini fallback, so annotation quality depends on closed-source judges.
- DPO causes a noticeable drop in some general benchmarks such as POPE.
- Evaluation relies heavily on GPT-4/GPT-4V judgments and a self-constructed HarmEval/HelpEval set.
Experimental setup
Datasets: SPA-VL training set LAION-5B Anthropic-Helpful AdvBench MM-SafetyBench
Benchmarks: HarmEval AdvBench MM-SafetyBench Anthropic-Helpful HelpEval POPE VQAv2 GQA VizWiz VQA ScienceQA TextVQA SEED-Bench
Metrics: unsafe rate harm score attack success rate win rate exact match F1
Baselines: LLaVA-1.5-7B InstructBLIP-7B LAMM-13B LLaMA-Adapter-v2-7B MiniGPT-4-7B mPLUG-Owl-7B Otter-9B InternVL-Chat-7B CogVLM2-LLama3-Chat-19B LLaVA-v1.6-34B QwenVL-Chat-7B InternLMXComposer-7B
Builds on: DPO PPO LLaVA-1.5 MD-Judge LAION-5B
Closest papers by meaning
- ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.92)
- Cross-Modal Safety Mechanism Transfer in Large Vision-Language Models (ICLR 2025 · similarity 0.91)
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.91)
- Pragma-VL: Towards a Pragmatic Arbitration of Safety and Helpfulness in MLLMs (ICLR 2026 · similarity 0.91)
- VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety (ICLR 2026 · similarity 0.91)
- Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.