Research map / Vision-language and multimodal / Multimodal LLMs
Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Safe RLHF-V, a multimodal safety alignment framework with the BeaverTails-V dual-preference dataset, the Beaver-Guard-V multi-level guardrail, and a Lagrangian constrained RL method with a budget-bounded multiplier update.
Problem
Safety alignment of multimodal LLMs while preserving helpfulness; existing multimodal safety preference data lacks disentangled helpfulness/safety labels and text-image dependence, and constrained optimization's stability in the multimodal setting is unclear.
Approach
Decouple helpfulness and safety preferences in a multimodal dataset with graded severity labels, train separate reward and cost models, and optimize the policy with PPO under a Lagrangian constraint whose multiplier is bounded by a budget bound for stability.
Key results
- Safe RLHF-V improves Beavertails-V safety win rate of LLaVA-1.5-7B from 0.5 to 0.579 and helpfulness to 0.770, and on MM-SafetyBench safety reaches 0.720.
- Beaver-Guard-V multi-level achieves 0.85 accuracy and 0.88 F1 in the binary-safety setting versus 0.64 accuracy for Llama-Guard-3-11B-Vision.
- The reward model reaches 82.2% accuracy with 5K preference pairs, while the cost model reaches 54.5% at 5K and 79.3% at 15K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dual helpfulness and safety preferences with constrained Safe RLHF-V instead of the unaligned model | LLaVA-1.5-7B · Beavertails-V | Safety win rate: 0.5 -> 0.579 | Helped |
| Use dual helpfulness and safety preferences with constrained Safe RLHF-V instead of the unaligned model | LLaVA-1.5-7B · Beavertails-V | Helpful win rate: 0.5 -> 0.770 | Helped |
| Use a dynamic update mechanism for the Lagrange multiplier rather than a fixed multiplier (ablation) | Safe RLHF-V on Qwen2-VL-7B · BeaverTails-V evaluation | Safety and helpfulness win rate | Helped |
| Increase RM-V training preference data from 5k to 30K (ablation) | RM-V · Reward model preference data | Accuracy: 82.2% -> 86.3% | Helped |
| Increase CM-V training preference data from 5k to 15k (ablation) | CM-V · Cost model preference data | Accuracy: 54.5% -> 79.3% | Helped |
| Increase CM-V training preference data from 20k to 25k (ablation) | CM-V · Cost model preference data | Accuracy: 79.9% -> 79.7% | Hurt |
Surprising findings
- The cost model accuracy is only 54.5% with 5K safety preference pairs, indicating difficulty of modeling safety constraints with limited data.
- Fixed Lagrange multipliers show performance that is parabolic in the initial value, with collapse at extremes, so dynamic updates were needed.
Limitations
- Study focuses on image-text scenarios; generalization to other modalities is untested.
- Cost model accuracy is low with small safety preference datasets.
- Evaluation relies on GPT-4o as judge with no human evaluation.
Experimental setup
Datasets: BeaverTails-V SPA-VL MM-SafetyBench Align-Anything-TI2T-Instruction-100K VLGuard VLSBench
Benchmarks: BeaverTails-V MM-SafetyBench SPA-VL VLGuard VLSBench
Metrics: win rate attack success rate accuracy F1-score false positive rate
Baselines: RLHF-V LLaVA-RLHF DPO PPO MM-RLHF Llama-Guard-3-11B-Vision
Builds on: Safe RLHF RLHF-V LLaVA-RLHF DPO PPO
Closest papers by meaning
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.93)
- Pragma-VL: Towards a Pragmatic Arbitration of Safety and Helpfulness in MLLMs (ICLR 2026 · similarity 0.92)
- When Safe Unimodal Inputs Collide: Optimizing Reasoning Chains for Cross-Modal Safety in Multimodal Large Language Models (AAAI 2026 · similarity 0.92)
- Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? (CVPR 2025 · similarity 0.91)
- SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization (CVPR 2026 · similarity 0.90)
- AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.