Research map / Vision-language and multimodal / Multimodal LLMs
Teach to Reason Safely: Policy-Guided Safety Tuning for MLRMs
Jingyu Zhang, Kun Yang, Ming Wen, Zhuoer Xu, Zeyang Sha, shiwen cui, Zhaohui Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes the safety-reasoning trade-off in multimodal large reasoning models (MLRMs) and proposes policy-guided safety tuning to restore safety without sacrificing reasoning.
Problem
Reasoning-based multimodal models are more prone to generating harmful content, so reasoning gains come with safety degradation.
Approach
Large-scale analysis identifying two mechanisms of safety degradation in MLRMs (the first being visual-related; the abstract is truncated) and a policy-guided safety tuning approach to teach models to reason safely. Details beyond the abstract are unknown.
Key results
- Reasoning-based MLRMs are shown to be more prone to generating harmful content than their non-reasoning counterparts, with safety performance degrading as reasoning improves (abstract finding; no specific numbers stated).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reasoning-oriented tuning of base MLLMs | Qwen2.5-VL-7B to R1-Onevision · BeaverTails-V | Harmful Rate (%): 58.40 -> 78.61 | Hurt |
| add explicit multi-step reasoning generation (ablation) | multimodal models · safety benchmarks | Harmful Rate | Hurt |
| apply PST (policy-guided SFT + SRPO) | R1-Onevision · BeaverTails-V | Harmful Rate (%): 78.61 -> 9.00 | Improved |
| add SRPO preference stage after PST-SFT (ablation) | R1-Onevision + PST-SFT · MM-SafetyBench | Harmful Rate (%): 5.48 -> 2.68 | Improved |
| add SRPO preference stage after PST-SFT (ablation) | R1-Onevision + PST-SFT · SIUO | Harmful Rate (%): 10.18 -> 12.57 | Hurt |
| add SRPO preference stage after PST-SFT (ablation) | LLaMA-CoT + PST-SFT · MMSafetyAware (over-sensitivity) | Refusal Rate (%): 54.91 -> 46.26 | Improved |
| use PST data instead of MSR-Align for alignment | R1-Onevision · MMSafetyAware (over-sensitivity) | Refusal Rate (%): 86.45 -> 69.39 | Improved |
| use PST instead of MSR-Align on general reasoning | R1-Onevision · VQAv2 | accuracy: 74.40 -> 80.87 | Improved |
| use PST instead of Think-in-Safety on LLaMA-CoT | LLaMA-CoT · ScienceQA | accuracy: 33.00 -> 84.10 | Improved |
| apply PST to LLaMA-CoT | LLaMA-CoT · GQA | accuracy: 60.00 -> 48.00 | Hurt |
| increase PST-SFT data from 1k to 2k-4k samples (ablation) | PST-SFT · BeaverTails-V, MM-SafetyBench, SPA-VL | Harmful Rate (%): 10.70 -> 11.54 (R1-Onevision, 1k -> 2k) | No clear effect |
| apply PST | R1-Onevision · BeaverTails-V failure types | Flawed Reasoning Initiation failure count: 331 -> 27 | Improved |
Surprising findings
- Reasoning capability gains come with a critical safety trade-off: reasoning-based models generate harmful content more readily.
Limitations
- Only the abstract was available; the full list of safety degradation mechanisms and method details are unknown.
Closest papers by meaning
- SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization (CVPR 2026 · similarity 0.92)
- When Safe Unimodal Inputs Collide: Optimizing Reasoning Chains for Cross-Modal Safety in Multimodal Large Language Models (AAAI 2026 · similarity 0.91)
- GuardTrace-VL: Detecting Unsafe Multimodel Reasoning via Iterative Safety Supervision (CVPR 2026 · similarity 0.91)
- Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models (ICLR 2026 · similarity 0.91)
- Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention (ICLR 2026 · similarity 0.91)
- Multimodal Situational Safety (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
