Research map / Large language models / Reasoning and chain of thought
Internalizing Safety Understanding in Large Reasoning Models via Verification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SInternal, which trains large reasoning models only on safety verification of their own sampled answers (with expert critiques from Claude-4-Sonnet), then optionally applies GRPO with a guard-model safety reward, improving robustness to out-of-domain and reasoning-specific jailbreaks.
Problem
Answer-centric safety alignment of large reasoning models optimizes final-answer compliance without building intrinsic ability to verify whether their own outputs are safe, leaving them vulnerable to jailbreaks and poorly generalizing.
Approach
Train LRMs with LoRA SFT exclusively on verification trajectories (critique plus safe/unsafe judgment) of their own sampled responses, produced by an expert model, rather than on safe answer trajectories; use this as initialization for GRPO with verifiable safety rewards.
Key results
- On DS-14B, SInternal reduces Fortress ASR from 52.6 (base) to 19.2 and WildJailbreak ASR from 44.4 to 6.8, while keeping XSTest compliance at 98.0.
- After GRPO, SInternal (+GRPO) on DS-14B lowers HCoT ASR to 62.0 vs 96.0 for base (+GRPO), the only method showing substantial HCoT defense.
- Verification F1 on DS-14B rises to 89.3 with SInternal versus 73.4 for GRPO alone and 71.1 for the base model.
Surprising findings
- Answer-centric alignment (STAR-1) lowered verification F1 for DS-7B to 32.2, below the random baseline of 62.4.
- Base model with GRPO alone outperforms SFT-based STAR-1 and SafeChain on several safety benchmarks, yet SInternal initialization yields still safer post-RL results.
Limitations
- Verification is restricted to post-generation checking; extending to dynamic self-verification during reasoning is left open.
- Verification capability still lags behind generation capability in the safety domain.
- Evaluation relies on Llama-Guard-3-8B as evaluator and a subset of 250 WildJailbreak prompts, and the expert model is a closed API model.
Experimental setup
Datasets: WildJailbreak STAR-1 DAPO
Benchmarks: StrongREJECT WildJailbreak Fortress Jailbreak-R1 HCoT Trotter XSTest MATH AIME2024
Metrics: attack success rate compliance rate pass@1 pass@16 F1
Baselines: SafeChain STAR-1 Base (+ GRPO) SafeChain (+ GRPO) STAR-1 (+ GRPO)
Builds on: GRPO STAR-1 SafeChain verl LLaMA-Factory
Closest papers by meaning
- Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models (NeurIPS 2026 · similarity 0.92)
- Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention (ICLR 2026 · similarity 0.92)
- AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models (ICLR 2026 · similarity 0.91)
- AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning (ICLR 2026 · similarity 0.91)
- SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment (NeurIPS 2025 · similarity 0.90)
- Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Away (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.